It doesn't work because you are taking the symmetric outer product between things which aren't the same. By direct computation with
$$\chi_3(\hat R)=\frac16\left\{\chi^3(\hat R)+2\chi(\hat R^3)\pm3\chi(\hat R^2)\chi(\hat R)\right\}$$
Using the character table for the symmetric group $S_3$ the symmetric triple product of $\Pi_u$ with itself does work out to $\Pi_u\oplus\Phi_u$.
$$\begin{align}\chi_3(E)&=\frac16\left\{2^3+2(2)+3(2)(2)\right\}=4\\
\chi_3(R(\theta))&=\frac16\left\{2^3\cos^3\theta+2(2\cos3\theta)+3(2\cos2\theta)(2\cos\theta)\right\}\\
&=\frac16\left\{6\cos\theta+2\cos3\theta+4\cos3\theta+6\cos2\theta+6\cos3\theta\right\}\\
&=2\cos\theta+2\cos3\theta\\
\chi_3(i)&=\frac16\left\{(-2)^3+2(-2)+3(2)(-2)\right\}=-4\\
\chi_3(C_{2d})&=\frac16\left\{0^2+2(0)+3(2)(0)\right\}=0\end{align}$$
I seem to recall doing this in stages as your attempt suggests but I was keeping around all the representations of the symmetric group at the intermediate steps. But when you need $\chi(\hat R^3)$ to compute a symmetric product between $\Delta_g$ and $\Pi_u$ which representation's $\chi(\hat R^3)$ do you choose?
EDIT: There is an extensive literature on the symmetric group but I am afraid to send you off into the tall grass when you just need a sliver of it all.
So maybe a mini-lecture on the symmetric group will suffice.
The set of operators that permute the elements of a set of $n$ identical objects is called the symmetric group $S_n$.
One possible notation for an element of $S_n$ is to write out the mapping like this:
$$R=\begin{bmatrix}1&2&3&4&5&6\\6&3&5&4&2&1\end{bmatrix}$$
Which means that under the mapping $R$, $1$ maps to $6$, $2$ maps to $3$, $3$ maps to $5$, $4$ maps to $4$, $5$ maps to $2$, and $6$ maps to $1$.
But more commonly seen is the cycle notation. Since $1$ maps to $6$ which maps back to $1$, that is written as the $2$-cycle $(12)$.
Then $2$ maps to $3$ maps to $5$ and finally back to $2$ giving us the $3$-cycle $(235)$.
$4$ maps to itself, the $1$-cycle $(4)$. So we can write the group element as a product of disjoint cycles $R=(235)(16)(4)$.
Normally the $1$-cycles are omitted for brevity but since they are a critical part of the formula we are trying to develop, we will continue to write them out.
Now we can see the effect on wave functions. Suppose we have a product of function of the wave functions of $3$ electrons
$$\psi=f(e_1)g(e_2)h(e_3)$$
We can operate on this with an operator that permutes its elements:
$$\hat O((123))f(e_1)g(e_2)h(e_3)=f(e_2)g(e_3)h(e_1)$$
Where each electron was replaced by its image under the mapping. If we the hit it with another operator
$$\hat O((12)(3))f(e_2)g(e_3)h(e_1)=f(e_1)g(e_3)h(e_2)$$
We could have computed the product of the $2$ permutations as
$$(12)(3)(123)=(1)(23)$$
To work out a product like this you go through the cycles starting from the rightmost (the first one to act) and trace what happens to each number as it is permuted by any applicable cycles.
We could have equivalently operated on the original product wave function with the product operator to get
$$\hat O((1)(23))f(e_1)g(e_2)h(e_3)=f(e_1)g(e_3)h(e_2)$$
Thus we see that the operators of $S_n$ form a group with the group product being functional composition, so it's always associative.
If you shuffle a deck of cards and then shuffle it again what you end up with is still a shuffled deck of cards so the set of permutation operators is closed under group multiplication.
There is an identity element, the permutation that leaves everything undisturbed, and an inverse like if some undergraduate comes into your office and messes up your neocube you can always put it back in order
(bad example, just bitching). So the first thing to do is to find the columns of its character table, the conjugacy classes. As an example, can we map
$(235)(16)(4)$ into, say, $(154)(36)(2)$ by conjugation? Let's write them out, one above anther:
$$\begin{align}(235)(16)(4)\\(154)(36)(2)\end{align}$$
If we mapped $1$ into $2$, $5$ into $3$, $4$ into $5$, $3$ into $1$, $6$ into $6$ and $2$ into $4$
$$\begin{bmatrix}1&2&3&4&5&6\\2&4&1&5&3&6\end{bmatrix}=(12453)(6)$$
Then everything would be aligned for the cycles to do the right things to the mapped elements, so we operate and then use the inverse to get
$$(13542)(6)(235)(16)(4)(12453)(6)=(154)(2)(36)$$
Just like we said. Hopefully you can see from this example that operators with the same cycle structure are conjugate and those with different cycle structure are not.
We can work out the number of elements in a conjugacy class
$$N\left(\left\{\mathop{\LARGE\Lambda}_{k=1}^nn_k\right\}\right)=\frac{n!}{\prod\limits_{k=1}^nk^{n_k}n_k!}$$
Where $n_k$ is the number of $k$-cycles in an element of the conjugacy class.
Now that we have the column headers of the character table of $S_n$ I have to admit that the rows are much more intimidating, but we only need two of them.
The first and easiest is the totally symmetric representation where each element is represented by the $1\times1$ identity matrix $\begin{bmatrix}1\end{bmatrix}$.
The is the representation that Bosons belong to. To understand the Fermion representation we have to explain what is an even and an odd permutation.
If you multiply the product of $2$ disjoint cycles on the left by a $2$-cycle (a transposition) that contains one element from each cycle you get $1$ long cycle:
$$(15)(123)(456)=(123564)$$
If you multiply a cycle by a transposition that contains $2$ elements of that cycle you get $2$ disjoint cycles:
$$(15)(123564)=(123)(456)$$
Thus every transposition you multiply a permutation by changes its number of disjoint cycles by $\pm1$.
For $S_n$ if $n$ and the number of disjoint cycles of an element $R$ of $S_n$ are both even or both odd, then the element $R$ is an even permutation and is the product of an even number of transpositions.
If $n$ is even and the number of disjoint cycles of $R$ is odd, or $n$ is odd and the number of disjoint cycles or $R$ is even, the $R$ is an odd permutation and is the product of an odd number of transpositions.
Even transpositions as a set are closed under group muliplication so they form a subgroup of $S_n$. There are as many odd permutations as even because $(12)$ times any even permutation is an odd permutation.
Also there are as many even permutations as odd because $(12)$ times an odd permutation is an even permutation.
So if the even permutations are represented by $\begin{bmatrix}1\end{bmatrix}$ and the odd permutations by $\begin{bmatrix}-1\end{bmatrix}$, that's the antisymmetric, or Fermion representation of $S_n$.
Now that we have all the columns and both of the rows of the character table of $S_n$ that we need we can use the orthogonality theorem to project out subspaces that belong to these important irreducible representations.
The next point I admit to being a little fuzzy on.
If each of the $n$ factor wave functions of a product wave function belongs to irreducible representation $\Gamma_r$ of point group $G$ then the trace of the operation $\pi$ of the group $S_n$ acting the operation $R\in G$ on the subspace of all product wave functions of all partners of the irreducible representation $\Gamma_r$ is
$$\chi(\pi)=\prod_{k=1}^n\chi_r^{n_k}(R^k)$$
Where $\pi$ has $n_k$ cycles of length $k$. So you just have to sum
$$\chi_s(R)=\frac1{n!}\sum_{C\subseteq S_n}\chi_s(C)\frac{n!}{\prod\limits_{k=1}^nk^{n_k}n_k!}\prod_{k=1}^n\chi_r^{n_k}(R^k)$$
Whew! Now you know why they didn't teach you that in general chemistry.
As an example, suppose we had $3$ factor wave functions belonging to the $T_{1u}$ representation of $O_h$. We need the character table of $O_h$
$$\begin{array}{c|ccccc|ccccc}O_h&E&8C_3&3C_4^2&6C_4&6C_{2d}&i&8S_6&3\sigma_h&4S_4&6\sigma_d\\\hline
A_{1g}&1&1&1&1&1&1&1&1&1&1\\
A_{2g}&1&1&1&-1&-1&1&1&1&-1&-1\\
E_g&2&-1&2&0&0&2&-1&2&0&0\\
T_{1g}&3&0&-1&1&-1&3&0&-1&1&-1\\
T_{2g}&3&0&-1&-1&1&3&0&-1&-1&1\\\hline
A_{1u}&1&1&1&1&1&-1&-1&-1&-1&-1\\
A_{2u}&1&1&1&-1&-1&-1&-1&-1&1&1\\
E_u&2&-1&2&0&0&-2&1&-2&0&0\\
T_{1u}&3&0&-1&1&-1&-3&0&1&-1&1\\
T_{2u}&3&0&-1&-1&1&-3&0&1&1&-1\end{array}$$
And although we don't actually need it, just to prove a point we provide the character table for $S_3$
$$\begin{array}{c|ccc}S_3&(1^3)&2(3)&3(21)\\\hline
(3)&1&1&1\\
(21)&2&-1&0\\
(1^3)&1&1&-1\end{array}$$
So we're ready to project into the irreducible representations of $S_3$
$$\begin{array}{c|ccccc|ccccc|l}O_h&E&8C_3&3C_4^2&6C_4&6C_{2d}&i&8S_6&3\sigma_h&4S_4&6\sigma_d&\Gamma\\\hline
(3)&10&1&-2&0&-2&-10&-1&2&0&2&A_{2u}+2T_{1u}+T_{2u}\\
(21)&8&-1&0&0&0&-8&1&0&0&0&E_u+T_{1u}+T_{2u}\\
(1^3)&1&1&1&1&1&-1&-1&-1&-1&-1&A_{1u}
\end{array}$$
As a sample calculation to get the $(21)\times3C_4^2$ entry of the above we saw that $(C_4^2)^3$ is in the same conjugacy class as $C_4^2$ and $(C_4^2)^2=E$ so the entry was
$$\frac16\left\{2\chi_{T_{1u}}^3(C_4^2)+(-1)2\chi_{T_{1u}}(C_4^2)+(0)3\chi_{T_{1u}}(C_4^2)\chi_{T_{1u}}(E)\right\}=\frac16\left\{2(-1)^3+(-1)2(-1)+(0)3(-1)(3)\right\}=0$$
It can be seen that we got $4$ boson irreducible representations and $1$ Fermion one. Also we found $3$ $(21)$ irreducible representations which are neither Fermion nor boson so it shows that you can't just subtract off the Fermion representations and always get the boson ones.
Well it's getting late... can I just say they lived happily ever after and it's time for bed?
