Which algebraic expression is correct 1 or 2? or both are wrong? 

I think the second equation is wrong. Cause, that should be
$$\frac{1}{r}=\frac{mk}{l^2}(1+\sqrt{1+\frac{2El^2}{mk^2}} cos(\theta'-\theta))\tag{1}$$
Or,
$$-\frac{1}{r}=\frac{mk}{l^2}(-1+\sqrt{1+\frac{2El^2}{mk^2}} cos(\theta-\theta'))\tag{2}$$
It's simple math I don't think I missed anything.
My attempt :
$$-\frac{\frac{l^2u}{mk}-1}{\sqrt{1+\frac{El^2}{mk^2}}}=\cos(\theta-\theta')$$
Wrong?
$$-\frac{l^2u}{mk}+1=\sqrt{1+\frac{2El^2}{mk^2}}\cos(\theta-\theta')$$
Then simple.
 A: From the first expression, the second expression follows. In the first step of your solution, you make a mistake with a minus sign. Note that your expression (1) is actually equivalent to expression 3.55 in the book. This follows from the equation $\cos(z)=\cos(-z)$ for all $z$, which implies that $\cos(\theta-\theta')=\cos(\theta'-\theta)$.
I shall give a derivation of expression 3.55:
The first expression in the book implies that $$\theta-\theta'=-\arccos\left(\frac{\frac{l^2u}{mk}-1}{\sqrt{1+\frac{2El^2}{mk^2}}}\right).$$
Taking the cosine of both sides, we obtain that
$$\cos(\theta-\theta')=\cos\left(-\arccos\left(\frac{\frac{l^2u}{mk}-1}{\sqrt{1+\frac{2El^2}{mk^2}}}\right)\right).$$
Using the equation $\cos(z)=\cos(-z)$ for the right-hand side, we see that
$$\cos(\theta-\theta')=\cos\left(\arccos\left(\frac{\frac{l^2u}{mk}-1}{\sqrt{1+\frac{2El^2}{mk^2}}}\right)\right).$$
Therefore,
$$\cos(\theta-\theta')=\frac{\frac{l^2u}{mk}-1}{\sqrt{1+\frac{2El^2}{mk^2}}}.$$
Next, we obtain that
$$\frac{l^2u}{mk}-1=\sqrt{1+\frac{2El^2}{mk^2}}\cos(\theta-\theta').$$
As you said, the rest is simple.
