Finding barycentric coordinates of a point $p$ in a triangle. Suppose you have three points $a,b,c \in \mathbb{R}^3$, not collinear. Let $p$ belonging to the triangle formed by $a,b,c$. For $x,y \in \mathbb{R}^3$ we define:
$$
\begin{array}{l}
P_{y}(x) = \left\langle x,\frac{y}{\lVert y \rVert} \right\rangle \frac{y}{\lVert y \rVert} \\
O_{y}(x) = x - P_{y}(x)
\end{array}
$$
Since $p$ belongs to the plane spanned by $a,b,c$ the following equation hold
$$
p = a + \alpha(b - a) + \beta(c - a) \Rightarrow p - a = \alpha(b - a) + \beta(c - a)
$$
My goal is finding $\alpha,\beta$. I'll do the calculations for $\alpha$, because $\beta$ is similar. I observe that
$$
\langle x,O_{x}(y) \rangle = 0
$$
Therofore in the plane equation performing the dot product with $O_{c-a}(b-a)$ leads to
$$
\langle p - a, O_{c-a}(b-a) \rangle = \langle \alpha(b - a) + \beta(c - a), O_{c-a}(b-a) \rangle \Rightarrow 
\langle p - a, O_{c-a}(b-a) \rangle = \alpha \langle b - a , O_{c-a}(b-a) \rangle \Rightarrow \alpha = \frac{\langle p - a, O_{c-a}(b-a) \rangle}{\langle b - a , O_{c-a}(b-a) \rangle}.
$$
With the same math I get
$$
\beta = \frac{\langle p - a, O_{b-a}(c-a) \rangle}{\langle c - a , O_{b-a}(c-a) \rangle}
$$
Is this correct?
 A: You’ve indeed come up with valid formulas for normalized barycentric coordinates relative to $\triangle{abc}$. To see this, we can use the fact that these coordinates can be interpreted as ratios of areas, i.e., the barycentric coordinates of a point $p$ that’s inside or on the triangle are $$[m_a:m_b:m_c]=\frac1{\operatorname{area}(\triangle{abc})}\left[\operatorname{area}(\triangle{pbc}):\operatorname{area}(\triangle{pca}):\operatorname{area}(\triangle{pab})\right].$$ (This also holds for exterior points, but you must use signed areas and take some care with the order of the vertices.)  
Your operator $O_y(x)$ computes the orthogonal rejection of $x$ from $y$, so $O_{c-a}(x)$ is the altitude from $x$ to the side $\overline{ac}$ of the triangle. Thus, $$m_b={\frac12\|O_{c-a}(p-a)\|\,\|c-a\| \over \frac12\|O_{c-a}(b-a)\|\,\|c-a\|}={\|O_{c-a}(p-a)\| \over \|O_{c-a}(b-a)\|}$$ and similarly for $m_c$. On the other hand, $\langle x,O_{c-a}(b-a)\rangle = \|x\|\,\|O_{c-a}(b-a)\|\cos\theta$, but $\|x\|\cos\theta$ is just the length of the orthogonal projection of $x$ onto $O_{c-a}(b-a)$, which is the same as the orthogonal rejection of $x$ from $c-a$, $O_{c-a}(x)$, so $$\langle x,O_{c-a}(b-a)\rangle = \|O_{c-a}(x)\|\,\|O_{c-a}(b-a)\|.$$ With this identity, your expression for $\alpha$ becomes $${\langle p-a, O_{c-a}(b-a) \rangle \over \langle b-a , O_{c-a}(b-a) \rangle} = {\|O_{c-a}(p-a)\|\,\|O_{c-a}(b-a)\| \over \|O_{c-a}(b-a)\|\,\|O_{c-a}(b-a)\|} = {\|O_{c-a}(p-a)\| \over \|O_{c-a}(b-a)\|}.$$  
We can also compare your expressions to a direct method of computing triangular barycentric coordinates via a linear transformation. W.l.o.g. we can examine this in $\mathbb R^2$. The normalized barycentric coordinates $m_b$ and $m_c$ of a point $p$ are just the coordinates of $p$ in a coordinate system that has $a$ as its origin and $c-a$ and $b-a$ as its bases. To simplify the calculations, we translate the standard coordinate system so that $a$ is at the origin. We then have $$\begin{bmatrix}m_b\\m_c\end{bmatrix} = \begin{bmatrix}x_b&x_c\\y_b&y_c\end{bmatrix}^{-1}\begin{bmatrix}x_p\\y_p\end{bmatrix} = \begin{bmatrix}{x_p y_c-y_p x_c \over x_b y_c-y_b x_c} \\ -{x_p y_b-y_p x_b \over x_b y_c-y_b x_c} \end{bmatrix}.$$ We also have (after some algebraic manipulation) $$\langle p,O_c(b)\rangle = {(x_p y_c-y_p x_c)(x_b y_c-y_b x_c) \over x_c^2+y_c^2}$$ and $$\langle b,O_c(b)\rangle = {(x_b y_c-y_b x_c)^2 \over x_c^2+y_c^2},$$ hence $$\alpha = {\langle p,O_c(b)\rangle \over \langle b,O_c(b)\rangle} = {x_p y_c-y_p x_c \over x_b y_c-y_b x_c} = m_b.$$ A similar computation verifies that $\beta=m_c$. Note that the above expression for $m_b$ can be written as $${\det{\begin{bmatrix}x_p&x_c\\y_p&y_c\end{bmatrix}} \over \det{\begin{bmatrix}x_b&x_c\\y_b&y_c\end{bmatrix}}},$$ which highlights the connection of barycentric coordinates to areas.  
As a third way to verify these formulas, we can compute the projections of $p$ onto the lines $\overline{ab}$ and $\overline{ac}$ parallel to $\overline{ac}$ and $\overline{ab}$, respectively. These projections can be computed by intersecting lines. Again, we work in $\mathbb R^2$ and place the origin at $a$ for simplicity. The projection onto $b$ parallel to $c$ is easily computed using homogeneous coordinates: $$\begin{align} ((x_b,y_b,1)\times(0,0,1))\times((x_p,y_p,1)\times(x_c,y_c,0)) &= (y_b,-x_b,0)\times(-y_c,x_c,x_p y_c-y_p x_c) \\
&= \left(-x_b(x_p y_c-y_p x_c), -y_b(x_p y_c-y_p x_c), x_c y_b-y_c x_b\right). \end{align}$$ Converted back to Cartesian coordinates, this is $$\left({x_b(x_p y_c-y_p x_c) \over x_b y_c-y_b x_c}, {y_b(x_p y_c-y_p x_c) \over x_b y_c-y_b x_c}\right) = {x_p y_c-y_p x_c \over x_b y_c-y_b x_c}b = \alpha b.$$ A similar computation shows that the projection onto $\overline{ac}$ is equal to $\beta c$, as desired.
A: If you need barycentric coordinates of a point, there is a neat formula
$$
\vec{\Lambda} = (-1)
\frac{
    \det
    \begin{pmatrix}
        0 & \vec{e}_1 & \vec{e}_2 & \vec{e}_3 \\
            p_x & a_x & b_x & c_x \\
            p_y & a_y & b_y & c_y \\
            1   &  1  & 1   & 1
    \end{pmatrix}
}{
    \det
    \begin{pmatrix}
        a_x & b_x & c_x \\
        a_y & b_y & c_y \\
         1  & 1   & 1
    \end{pmatrix}
}
$$
Here $\vec{e}_1$, $\vec{e}_2$, and $\vec{e}_3$ are some formal orthonormal vectors. Components of "vector" $\vec{\Lambda}$ will be equal to barycentric coordinates.
EXAMPLE
Let $a = (-2;-1)$, $b = (3; -1)$, and $c = (1;4)$. I want to calculate barycentric coordinates of $p = (2;1)$. The equation above gives
$$
\vec{\Lambda} = (-1)
\frac{
    \det
    \begin{pmatrix}
        0 & \vec{e}_1 & \vec{e}_2 & \vec{e}_3 \\
            2 & -2 & 3  & 1 \\
            1 & -1 & -1 & 4 \\
            1 &  1 & 1  & 1
    \end{pmatrix}
}{
    \det
    \begin{pmatrix}
        -2 & 3  & 1 \\
        -1 & -1 & 4 \\
         1 & 1  & 1
    \end{pmatrix}
} = \frac{1}{25} \vec{e}_1 + \frac{14}{25} \vec{e}_2 + \frac{2}{5}\vec{e}_3,
$$
thus barycentric coordinates of $p$ are $1/25$, $14/25$, and $2/5$.
REFERENCES
This method together with theory it's based on is presented by its authors in "Workbook on mapping simplexes affinely" and "Beginner's guide to mapping simplexes affinely". You may want to check this references for more details.
