Game with Logit Chance of Winning Two players (player $A$ and $B$) play a game. Let $a$ denote $A's$ effort and $b$ denote $B's$ effort. Suppose that the chance of winning in this game is a logistic function of effort (i.e. is a function of the difference between the amount of effort exerted):
$$p(win_a)=\frac{e^{a}}{(e^{a}+e^{b})}$$
Suppose the winner receives some prize $P$. Effort costs $c$ per unit for both players. A will maximize
$$argmax_a\bigg(p\frac{e^{a}}{(e^{a}+e^{b})}-ca\bigg)$$
Taking the derivatives yields
$$p\frac{e^{a+b}}{(e^{a}+e^{b})^2}-c$$
What is A's choice of effort?

My attempt: Define $X:=e^a$ and $Y:=e^b$, then we have the condition $c=p\frac{XY}{(X+Y)^2}$.
Then we get that $$X=\frac{pY-2cY+Y\sqrt{p(p-4c)}}{2c},\:X=\frac{pY-2cY-Y\sqrt{p(p-4c)}}{2c}$$
However, this seems unsatisfactory because i) $p>4c$ for the radical to make sense ii) we get two different equations for $a$, and I'm not sure which to choose iii) $a$ seems to be strictly increasing in $Y$, meaning that the more effort $B$ exerts, the more effort $A$ wants to exert (which makes no sense).
 A: Don't forget the endpoints. 

If $p \le 4c,\;$then for fixed $b \ge 0,\;$the expected net profit for $A,\;$as a function of $a,\;$is strictly decreasing, hence the expected net profit for $A\;$is maximized when $a=0$. 

In other words, if $p \le 4c,\;$the prize is not worth the effort.

Next, assume $p > 4c$.

Letting $r ={\large{\frac{p}{4c}}},\;$we have $r > 1$.

For fixed $b \ge 0$, there is a unique value of $a > 0$ which maximizes $A$'s expected net profit, namely, the unique positive real number $a$ such that $e^a$ is the positive root of the quadratic equation you analyzed. 

Simplifying that solution,  when $b \ge 0$ is fixed and known, $A$'s optimal value for $a$ is the positive real number $a$ such that
$$e^a = e^b\left(2r-1+\sqrt{r(r-1)}\right)$$
Note that $r > 1$ implies $2r-1+\sqrt{r(r-1)} > 1$, hence, for fixed $b \ge 0$, $A$'s optimal value for $a$ will be such that $a > b$.

However, if the value of $b$ is unknown, there is no optimal pure strategy for $A$.

To see this, suppose $a=a_1$ is optimal for $A$. 

Then $B$ can choose $b=b_1$ such that
$$e^{b_1} = e^{a_1}\left(2r-1+\sqrt{r(r-1)}\right)$$
in which case, $b_1 > a_1$, hence $a=a_1$ is not optimal against $b=b_1$, contradiction.

Thus, optimal strategies for $A,B$, if they exist, will have to be mixed strategies.

However:

If $A,B\;$both agree to use zero effort, then $A\;$and $B\;$will each have expected net profit ${\large{\frac{p}{2}}}$.

But unless that agreement is "locked in", each player has an incentive to cheat.
A: This is a textbook problem in game theory. In fact, you can find a bunch of similar problems in Bolton, Patrick, and Mathias Dewatripont. Contract theory. MIT press, 2005.
For this one, I suggest the classical approach by comparing MR and MC in different cases carefully. 

first consider the case when $b>0$. the optimal effort of $A$ given $B$'s (and parameters $P,c$) is
$$
a^*(b; P,c)=\text{argmax}_{a\geq0} P\dfrac {e^a}{e^a+e^b}-ca
$$
the marginal cost of effort at any level is
$$
MC(a)=c
$$
the marginal return of effort at level $a$ is
$$
MR(a)=P\dfrac {e^{a+b}}{(e^a+e^b)^2}
$$
then 
$$\dfrac {dMR(a)}{da}=P\dfrac {e^{a+b}(e^b-e^a)}{(e^a+e^b)^3}\begin{cases}
>0,  & \text{when    }  0\leq a<b \\
=0, & \text{when    }  a=b \\
<0, & \text{when    }  a>b
\end{cases}$$
from this we can conclude that $MR$ increases from $MR(0)=Pe^b/(1+e^b)^2$ to $MR(b)=P/4$, and then decreases to $0$ as $a\rightarrow \infty$.
so $MR\in (0, P/4]$.
case 1
when $c> P/4$, $MC> MR$ for all $a$, so the optimal choice of effort is $0$.
case 2
when $c=P/4$, $MC=MR$ at one point $b$, but for all $a\neq b$, $MC>MR$, so the optimal choice is still $0$.
case 3
when $P/4>c\geq Pe^b/(1+e^b)^2$, there are two points where $MR=MC$, one less than $b$ and another greater than $b$. from our above discussion we know only the second one satisfies the second order condition, so our candidate is 
$$
\hat a=b+\ln(\dfrac {P-2c+\sqrt{P^2-4Pc}}{2c})
$$
note that zero effort is always possible, to choose $\hat a$, we must ensure
$$
P\dfrac {e^{\hat a}}{e^{\hat a}+e^b}-c\hat a\geq P\dfrac {1}{1+e^b}
$$
denote $c_0(P,b)$ as the critical value when the above relationship holds with equality. then for $c>c_0$, player A would still choose zero effort; for $c=c_0$, A is indifferent between choosing $0$ or $\hat a$, and for $c<c_0$, $\hat a$ is the unique optimal choice.
case 4
when $Pe^b/(1+e^b)^2>c>0$, $MC=MR$ at one point $\hat a=b$, and the second order condition also holds, then
$$
\hat a=b+\ln(\dfrac {P-2c+\sqrt{P^2-4Pc}}{2c})
$$
is the unique candidate. To become a ture optimizer, we also need $c<c_0$.
when $b=0$, the problem is trivial.

In summary, the best response correspondence of A given B's effort is
$$
a^*(b;P,c) =
\begin{cases}
\{0\}, & b=0, c\geq P/4\\
\{\ln(\dfrac {P-2c+\sqrt{P^2-4Pc}}{2c})\} & b=0, c< P/4\\
\{0\},  & b>0, c\geq P/4 \\
\{0\},  & b>0, c< P/4, c>c_0(P,b)\\
\{0,b+\ln(\dfrac {P-2c+\sqrt{P^2-4Pc}}{2c})\}, & b>0, c<P/4, c=c_0(P,b) \\
\{b+\ln(\dfrac {P-2c+\sqrt{P^2-4Pc}}{2c})\}, & b>0, c<P/4, c<c_0(P,b)
\end{cases}
$$
note that $a^*$ is not increasing in $b$, since when $b$ goes larger, the critical value of marginal cost must be lower to outperform zero-effort, and eventually lower than the true value of $c$. After that player A would switch to zero effort.
