Proof that Maximum Likelyhood Estimation Finds the True Distribution Parameters I don't have any Statistics background. I was learning Convex Optimization and MLE appeared. 
MLE is pretty easy and intuitive to understand: given some i.i.d. samples and knowing that those samples come from a distribution with unknown parameters, we want to find those parameters by maximizing the joint density function for the fixed samples. 
Then it occurred to me, why maximizing the joint density function, why not maximizing the least square of the PDFs? why not maximizing the L1 norm, L3 norm, the infinity norm of the PDFs?
After some digging, I did find proof that MLE finds the exact correct parameters of the distribution if the distribution is normal. But is there any proof that it finds the true parameters for ALL distributions? If not, why MLE? What is the best we can do in theory in finding those parameters?
It will be greatly appreciated if the answers could be in layman's term.
 A: Two issues.
Notations & terms

*

*rv: random variable

*iid: independent identical distribution

*pdf: probability density function

*jpdf: joint pdf

*$X_1,X_2,...,X_n$: $n$ iid

*$x_1,x_2,...,x_n$: an observation

*$\theta$: the wanted parameter(s)

*$\Theta$: parametric space (i.e. all the values $\theta$ may take)


Fact: the parameter ($\theta$) is a number but unknown, not a rv.

ISSUSE 1 MLE
The key to understand MLE is to think the same thing (jpdf) from other side, the PARAMETRIC SPACE side.
Namely, you take the PARAMETRIC SPACE as the domain of the jpdf.

Then, we denote this jpdf as $L(\theta;x_1,...,x_n), \theta\in\Theta.$

That is, for each possible $\theta_0$, there is a corresponding $L(\theta_0;x_1,...,x_n)$ which is a number because on the one hand $x_1,...,x_n$ are given, on the other hand $\theta$ is given.
You have known that the following step is maximizing this function.
Wait a minute. What is maximizing?
This word means that you choose the maximum possible (i.e. $L(\theta)$) case  among a bunch of cases (each $\theta$ in $\Theta$ representing a case).
Therefore, the cost function you choose is legal, if the maximizing of this cost function is equivalent to the maximizing of $L(\theta;x_1,...,x_n)$ among the cases $\theta\in\Theta$.
In fact, we often choose the cost function as $\ln L(\theta;x_1,...,x_n)$ because this is convenient for many distributions.

Theorem. Maximizing  $L(\theta)$ is equivalent to maximizing $\ln L(\theta)$, among all $\theta\in\Theta$.
Hints: It it obviously that $\ln x$ is monotone incresing in $x$ when $x>0$.
Meanwhile, $L(\theta)>0, \theta\in\Theta$.
Take $\theta_0\in\Theta$, s.t. $L(\theta_0)=\max_{\theta\in\Theta}L(\theta).$ Then $\ln L(\theta_0)=\max_{\theta\in\Theta}\ln L(\theta)$.

In fact, the MLE method is not the BEST. Consider the following example:

Consider $(X_1,..,X_n)\stackrel{iid}{\sim}N(\mu,\sigma^2)$.
We can calculate the MLE of $\sigma^2$ is $\hat\sigma^2=\dfrac1n\sum\limits_{i=1}^n(X_i-\bar X)^2$, which is obviously biased.

ISSUSE 2 Best way to estimate parameters of a distribution
On parameters of a distribution, there are point estimation and interval estimation.
What you mentioned is point estimation, in which the BEST estimator is called UMVUE (uniformly minimum variance unbiased estimator).
The MLE method is not a approach to get UMVUE directly (as we've shown that MLE is sometimes biased) but an intuitive one.
On UMVUE, there are too many theories. You may google Lehmann-Scheffe theorem (a way to find UMVUE) for more details.
