The minimum value of $P(A \cup B )$, if $P(\bar B)={ \{P(A \cup B)}\}^2$ What I tried: $$1-P(B)=\{P(A)+P(B)-P(A \cap B)\}^2$$
Now since $0\le P(B) \le1 \\
$:
$\therefore 1-P(B) \in\  [0,1]\\$
Also, the R.H.S is also the square value of a probability hence it is also in $[0,1]$.
Next, I decided to check the options given if any of them has the range $[0,1]$ and apparently there were two of them.
Looking at the options I got an intuition that the answer probably is the root of a quadratic equation but I am unable to form that quadratic equation.
(I will attach the image for reference)
I also tried using Venn diagram and had an idea that $1=\text{Probability of the sample space}$ and then I could write some expression for $P(\bar{B})$ by equating it with $P(A)-P(A\cap B)\  \mathbf{\text{along with something more}}$ that something more is what I am unable to find out. If I get that then I think I can let some terms as $x$ and form and equation.
I also made an approach using conditional probability but that didn't give any progress.
OR Maybe I am wrong all along.
This is the Question:
I know B can't be the answer A and C hint at it being a quadratic function but if I don't get an absolute value then, option D could also be correct.

 A: For notational simplicity, let
\begin{align*}
x&\equiv\mathbb P(A\setminus B),\\
y&\equiv\mathbb P(B).
\end{align*}
Then, we have
\begin{align*}
\mathbb P(A\cup B)=x+y,
\end{align*}
and the constraint $\mathbb P(\overline B)=[\mathbb P(A\cup B)]^2$ can be written as $1-y=(x+y)^2$ (drawing a Venn diagram may help).
Therefore, the problem can be restated as follows:
\begin{align*}
\text{minimize }x+y&\mathrel{\phantom=}\text{such that}\\
1-y&=(x+y)^2,\\
x&\geq0,\\
y&\geq0,\\
x+y&\leq 1.
\end{align*}
The condition $1-y=(x+y)^2$ is a quadratic equation. Solving for $x$ yields two roots (as a function of $y$), one of which is negative for any possible value of $y\in[0,1]$, and hence inadmissible. The only candidate for a non-negative root is $$x=\sqrt{1-y}-y.$$ Plugging this back into the minimization problem above yields the following simplified problem:
\begin{align*}
\text{minimize }\sqrt{1-y}&\mathrel{\phantom=}\text{such that}\\
\sqrt{1-y}&\geq y,\\
y&\geq0,\\
\sqrt{1-y}&\leq 1.
\end{align*}
If the first constraint were slack, that is, $\sqrt{1-y}>y$, then you could slightly increase $y$ and further decrease the value of the objective function $\sqrt{1-y}$. Therefore, the minimizing solution $y^*$ must solve $\sqrt{1-y^*}=y^*$, and the only number that satisfies this is $$y^*=\frac{\sqrt{5}-1}{2}.$$ This is also the minimum value we sought, since
\begin{align*}
\mathbb P(A\cup B)=x^*+y^*=\left(\sqrt{1-y^*}-y^*\right)+y^*=\sqrt{1-y^*}=y^*=\frac{\sqrt{5}-1}{2}.
\end{align*}
A: As P(AUB) is minimum so we will take intersection is maximum
so first case P(B) is greater then P(A) then u will get a quadratic equation and by solving it u will get option a ,
and coming to the second case P(A) greater than P(B)
, which is not possible as ,
By the given hint u get P(A) is greater than P(B) which is a controversy so option a is answer
