How can I avoid the wrong solution when squaring both sides of an equation? We have the equation, $\sqrt{x^2-1}=x-3$
So, squaring both sides we get, $x^2-1=x^2+9-6x$
Which gives, $x=\frac{5}{3}$
However, by verifying the solution one immediately realises this is a false solution. These equations are inconsistent with each other.
Now this makes me realise that:

*

*Either the way I am squaring both sides is wrong i.e., we have to assume some additional conditions when squaring both sides similar to when we divide by a variable we assume it is not equal to $0$ i.e., from $ax=0$ we don't get $a=0$, rather we get $x\neq0\Rightarrow a=0$. But, then I don't know those conditions.


*Or I am getting an alright result because I assumed that an inconsistent set of equations have a solution, which is quite similar to when we try to solve a pair of inconsistent linear equations in two variables and we get a contradictory result. The only difference is that we immediately realise the contradiction in the latter case because we get results like '$0=2$' but here we realise the contradiction only after verifying. This creates a bigger problem because it tells me that after solving any set of equations, I should verify the solution, which I am afraid to say that I mostly don't do.
So, is my method of squaring both sides right and I should verify all/some specific sets of equations I solve, Or there are some conditions I am missing while squaring both sides of an equation? If there are such conditions then what are they?
 A: I know you have a satisfactory solution already, but I thought I’d mention that if you consider the problem graphically, it is easy to see why there is no solution, and why squaring the equation creates a false solution.
The graph of $y=\sqrt{x^2-1}$ is the part of a rectangular hyperbola lying in the first quadrant only (since $x\geq3$), and this has an oblique asymptote which is the line $y=x$. The graph of $y=x-3$ is parallel to this asymptote, and lies and therefore remains below the hyperbola for all positive values of $x$. Therefore no solution.
However, by squaring both sides you are now effectively solving an additional equation, namely $$-\sqrt{x^2-1}=x-3$$ and this is where the extraneous solution comes from - because the LHS is that part of the same hyperbola in the fourth quadrant, with asymptote $y=-x$, and will therefore intersect with $y=x-3$.
A: Nothing new here, but shorter and (I hope) clearer.
It is certainly true that
$$
\sqrt A = B \;\;\Longrightarrow\;\;A=B^2
$$
But this says that the solutions of the left equation are a subset of the equation on the right. Saying this another way, there may be solutions of the right-hand equation that are not solutions of the left-hand one; these are often called “extraneous solutions”. If you find a solution of the right-hand equation, it won’t necessarily be a solution of the left one; you have to check by substitution.
A: 
We have the equation $\sqrt{x^2-1}=x-3.\tag1$

The is an inconsistent equation, that is, it has no solution.

squaring both sides gives $x=\frac53.\tag2$ this is a false solution.

This is an extraneous solution of equation $(1).$


*

*Either the way I am squaring both sides is wrong


Nope, that squaring step was perfectly valid: $$\forall x{\in}\mathbb R\; \Big((1){\implies}(2)\Big).$$



*Or I am getting an alright result, because I had gotten a contradictory result by assuming that an inconsistent set of equations has a solution


Yes. Here, I discussed another such example of deductive explosion.

This creates a bigger problem because it tells me that after solving any set of equations, I should verify the solution, which I am afraid to say that I mostly don't do.

It is natural to solve a given equation (let's call it A) by reasoning that if A, then B, then C, then D, where D (e.g., $x=-3,7$) contains the solutions. But, in fact, D contains merely the candidate solutions, and, in this approach, equation A technically is not solved until the candidate solutions are verified. I elaborated on this in this Answer, which contains tips for recognising when and how extraneous solutions may arise.
Alternatively, to avoid extraneous solutions while solving an equation, we could carefully ensure that every step is reversible (but this method is probably not worth teaching in class), like so: \begin{align}\forall x\in\mathbb R \Bigg[\quad\quad\quad&\sqrt{x^2-1}=x-3\\\iff {}& x^2-1=(x-3)^2 \quad\text{and}\quad x-3\ge0 \\\iff {}&x\in\emptyset\quad\Bigg].\end{align}
A: Understanding the general pattern at work: equational reasoning can obscure underlying logical structure
This kind of equational reasoning is a useful tool when applied appropriately, but many explorers find themselves, as in this question, confused as to how to ensure they are performing valid reasoning: 'I did only valid equational manipulations which always worked before, what went wrong?'
In particular, the folk equational reasoning consists of writing down equations and their manipulations without tracking the lines of implication between them. But there is an important perspective shift which transforms this tool from confusing to crystal clear - just look at the logical structure implicit in your equations!
A few general observations
What is a 'solution'?
To be a solution to an equation is to imply the equation (e.g. $x = 1 \implies x^2 = 1$ and $x=-1 \implies x^2 = 1$).
Note that the reverse is not necessarily true: if an equation has multiple solutions, it only implies that one or other solution is true. It doesn't imply each individually (unless the solution is unique).
$$x=1 \implies x^2 = 1$$
$$x^2 = 1 \quad \not \!\!\!\!\implies x=1$$
But we can say that an equation implies the disjunction (or) of all of its solutions, and vice versa
$$x^2 = 1 \iff (x = 1) \lor (x = -1) $$ (read $x = 1$ OR $x = -1$).
So we can see there's a difference between 'find a solution' and 'find the solution' and 'find all solutions' (aka 'solve'), though in natural language we sometimes equivocate between these.
What is an equational manipulation?
Applying a function to both sides of an equation leads to a valid implication of the equation, but the reverse implication is not necessarily valid. In logical terms, we have the schema
$$a = b \implies f(a) = f(b)$$
for any expressions $a, b$ and function/formula $f$.
Now, if there is an inverse $f^{-1}$ of $f$ (for example if $f$ is an addition or subtraction) then we do have the implication in the other direction because (by the schema)
$$f(a) = f(b) \implies f^{-1}(f(a)) = f^{-1}(f(b))$$
and the latter is the same as $a = b$ by the inverse relationship of $f$ and $f^{-1}$.
But if that inverse does not exist, we do not have the reverse implication.
Squaring
In the case of squaring equations, $f$ has $f(a) = a^2$. This function is not invertible in general (though it is if we already know $a > 0$).
A few other common cases

*

*Addition and subtraction always have inverses.

*Division (by nonzero) always has an inverse. If we divide by something that might be zero, we might draw invalid conclusions.

*Multiplication (by nonzero) always has an inverse. If we multiply by something that might be zero, it might not be invertible and we need to check (or assert it is not zero).

*Exponentiation always has an inverse.

*Logarithm (of nonzero) always has an inverse.

I analyze your 3 steps.

*

*$\sqrt{x^2 - 1} = x - 3$

*$x^2 - 1 = x^2 + 9 - 6x$

*$x = \frac 5 3$
Now certainly 3 $\implies$ 2. So 3 is one solution to 2.
But as we noted, squaring is not invertible. So although 1 (our target) $\implies$ 2, we do not have the reverse.
We can write (abusing notation a little)
1 $\implies$ 2 $\impliedby$ 3. But for 3 to be a solution to 1, we need 3 $\implies$ 1, which we don't have.
Candidate solutions
In general, we can often perform non-invertible transformations (like your squaring) to manipulate facts we know (like your target equation) into other facts we can deduce. If those other facts are in familiar or easy forms (like your nice quadratic) it can be a useful generator of candidate solutions which we might not have found otherwise.
But unless we have the backwards implication from candidate solution to target equation, we do not have a solution. So here's a general rule of thumb (which, importantly, always works!):
If the lines of implication between your equations don't point all the way from 'solution' to target, you have a candidate 'solution' and need to check. In equational reasoning, this is any time you apply a non-invertible transformation.
A: Here is an answer pitched at a lower level.
Question: How can I avoid the wrong solution when squaring both sides of an equation?
Answer: After solving the equation in the usual way, check each solution to see which ones fit the original equation.
If you also want to know where a wrong solution came from, then find the first line of your working where the solution fits. You will find that, in that step you eliminated some restriction on $x$, allowing the wrong solution to "sneack in".
EXAMPLE:
$\sqrt{x^2-1}=x-3$
$x^2-1=x^2-6x+9$
$6x=10$
$x=\frac{5}{3}$?
Plug this in to the original equation: it doesn't fit. So it's not a solution. (So the original equation has no solution.)
OK, but where did $x=\frac{5}{3}$ come from?
Let's see, the first line of my working where $x=\frac{5}{3}$ fits, is Line 2 ( $x^2-1=x^2-6x+9$ ).
So in that step, I must have eliminated some restriction on $x$.
Ah, yes: in Line 1 ( $\sqrt{x^2-1}=x-3$ ), we require $x^2-1\ge0$, but this restriction does not exist in Line 2.
So you can see how $x=\frac{3}{5}$ snuck in, in Line 2.
