If $x$, $y$ and $z$ are distinct positive integers and $x+y+z=11$ then what is the maximum value of $(xyz+xy+yz+zx)$? 
If $x$, $y$ and $z$ are distinct positive integers and $x+y+z=11$ then what is the maximum value of $(xyz+xy+yz+zx)$?

We know that product is maximum when difference between $x$, $y$ and $z$ is minimum.
So, I assumed $x=3$, $y=4$ and $z=4$. 
Now putting this value in $xyz+xy+yz+zx$ I got my answer $88$. But actual answer is $78$. Where am I doing it wrong?
 A: Note that $(x+1)(y+1)(z+1)=1+(x+y+z)+(xy+yx+zx)+xyz$ so the sum you want is $$(x+1)(y+1)(z+1)-12$$ which justifies your comment about the product. 
If $a\gt a-1\ge b+1\gt b$ we have $$(a-1)(b+1)=ab+(a-b)-1\gt ab$$ since $a-b\ge 2$. 
The best selection of distinct integers is (as others have noted) $5+4+2=11$. And the product formula gives $90-12=78$.
A: put $z=11-x-y$ in the equation and differentiate it with respect to $x$.
here are the values of $x$,$y$,$z$  for maximum value.
y=4.18   x=3.40   z=3.42  and maximum value is $77.11$.
However, I solved it using wolfram-math but you can solve it by following the instructions given.
However, my solution is for real numbers but you want it for integers.
A: the way the question is worded, $x$ and $y$ and $z$ cannot be equal (they must instead be distinct) so $y$ and $z$ cannot be $4$.
so the correct answer is choosing $5$, $4$ and $2$ for $x$, $y$ and $z$.
A: For $x=5$, $y=4$ and $z=2$ we get a value $78$.
We'll prove that it's a maximal value.
Indeed, let $x<y<z$, $x=1+a$, $y=2+a+b$ and $z=3+a+b+c$, 
where $a$, $b$ and $c$ are no-negative integer numbers. 
Thus, the condition gives
$$1+a+2+a+b+3+a+b+c=11$$ or
$$3a+2b+c=5,$$
which says that $a\in\{0,1\}$. 
Now, let $a=0$.
Thus, $x=1$, $y+z=10$ and we need to prove that
$$y+z+2yz\leq78$$ or
$$yz\leq68,$$
which follows from AM-GM:
$$yz\leq\left(\frac{y+z}{2}\right)^2=25<68.$$
Let $a=1$.
Hence, $x=2$, $y+z=9$, $z\geq5$ and we need to prove that
$$2y+2z+3yz\leq78$$ or
$$yz\leq20$$ or
$$(9-z)z\leq20$$ or
$$(z-4)(z-5)\geq0,$$
which is obvious.
By the same way we can prove that $42$ is a minimal value, 
where the equality occurs for $(x,y,z)=(1,2,8)$.
Indeed, if $a=0$ then we get $x=1$, $y\geq2$, $y+z=10$, $6\leq z\leq8$ and we need to prove that
$$xy+xz+yz+xyz\geq42$$ or
$$y+z+2yz\geq42$$ or
$$yz\geq16$$ or
$$(10z)z\geq16$$ or
$$(8-z)(z-2)\geq0,$$ which is obvious.
If $a=1$ then $x=2$, $y\geq3$, $y+z=9$, $5\leq z\leq6$ and we need to prove that 
$$2(y+z)+3yz\geq42$$ or
$$yz\geq8$$ or
$$(9-z)z\geq8$$ or
$$(8-z)(z-1)\geq0,$$
which is obvious.
Done!
A: $11$ is reasonably small. So we can search exhaustively through the options to find maximum $f=xyz+xy+yz+zx$.
Take $x<y<z$. Then smallest values for $x,y$ are $1,2$ and thus greatest $z=8$. Then we can work down values of $z$ and partition the residue accordingly, avoiding duplicates.
$\begin{array}{|c|c|} \hline
(x,y,z) & f\\ \hline
(1,2,8) & 42 \\
(1,3,7) & 52 \\
(1,4,6) & 58 \\
(2,3,6) & 72 \\
(2,4,5) & 78 \\ \hline
\end{array}$
