$O(3)/SO(3) \cong \mathbb{Z}_2$ Progress:
$\phi: O(3) \rightarrow \mathbb{Z}_2$
$\psi: O(3) \rightarrow O(3)/SO(3)$
$\theta: O(3)/SO(3) \rightarrow \mathbb{Z}_2$
$\phi(O) = \text{det} (O)$ with $O \in O(3)$, that way 
$\phi(O) \mapsto \{-1,1\} \cong \mathbb{Z}_2$, where 1 is the identity element.
Ker($\phi$) = $\{O \in SO(3) | \phi(O)=1\} = SO(3) $, since $\text{det}(O)=1$ for $O \in SO(3).$
By the multiplicative property of the determinant function, $\phi$ = homomorphism.

***What is the form of the canonical homomorphism ($\psi$) in this case?
I'm used to the coset language, 
i.e. $\psi: G \rightarrow G/\text{Ker}(\phi)$ with $\psi(g)=gK$ for $K=\text{ker}(\phi)$

If this were settled, then $\theta$ is an isomorphism by the isomorphism theorem.
 A: The mere fact that $O(3)/SO(3) \cong \mathbb{Z}_2$ does not imply that there even needs to be a map $\psi: O(3) \to SO(3)$; in general there aren't homomorphisms from non-abelian groups to their subgroups (or even normal subgroups). After all: how do we know where to send the elements in the big group that are not part of the small group?
That said: in this special case the map $\psi$ does exist. The point is that not only do we have $O(3)/SO(3) \cong \mathbb{Z}_2$ but we even have:
$$O(3) \cong SO(3) \times \mathbb{Z}_2$$ 
which means that indeed we do have four maps, your $\phi$ and $\psi$ pointing out of $O(3)$ onto its factors and in the other direction we have embeddings $SO(3) \to O(3)$ and $\mathbb{Z}_2 \to O(3)$.
The decomposition $O(3) \cong SO(3) \times \mathbb{Z}_2$ can be seen really concretely:
every determinant 1 matrix $A$ in $O(3)$ corresponds to the pair $(A, 1)$ in $SO(3) \times \mathbb{Z}_2$ and for every $B \in O(3)$ with determinant minus one we have that $-B$ lives in $SO(3)$ and we let $B$ correspond with the pair $(-B, -1)$ in $SO(3) \times \mathbb{Z}_2$. 
Now I leave it to you to get the projection operator $\psi$ out of this picture.
EDIT: when it comes to $\theta$: as the comments say you only need to understand $\phi$ to get that $\theta$ is an isomorphism by the isomorphism theorem; this part works also in cases where the big group is not a direct product of the smaller group and the quotient (and $\psi$ does not exist).
EDIT 2: I thought seeing an example where there is no map $\psi$ would help you clarify this issue. Here is one: take the dihedral group $D_4$ with eight elements, so the symmetry group of a square. This is the big group, similar to $O(3)$ in your example. Take $C_4$ the cyclic group of four elements as the small group. You can write down $\phi$ and $\theta$, the quotient is again the two element group. Also thinking about $D_4$ as symmetries of the square, there is a nice geometric analogy with your case: the small group consists of rotations only while the big group consists of rotations AND reflections. However in this case there is no homomorphism from the big group to the small one.
A: I believe the codomain of $\psi$ is wrong, it should be $O(3)/SO(3)$ (which is isomorphic to $\mathbb Z_2$). Then the form is precisely as you described. More concretely,
$\varphi(O) = SO(3)$ if $\det O = 1$, and $\varphi(O) = O(3) \setminus SO(3)$ otherwise. Note that $O(3) \setminus SO(3)$ is actually equal to $(-I)\cdot SO(3)$.
