Might $a(x)$ be irreducible in $F[x]$ but reducible in $F[x]/\langle p(x)\rangle$, where $a(x)$ is not a multiple of $p(x)$? Might $a(x)$ be irreducible in $F[x]$ but reducible in $F[x]/\langle p(x)\rangle$, where $a(x)$ is not a multiple of $p(x)$?
$F[x]/\langle p(x)\rangle $ does not have to be a field. Could you name an example, preferably one for a field another for a non-field just so I can see the difference?
It was suggested by Arturo that I should re-edit this question since it was not clear enough:
Let F be a field, and say $a(x)∈F[x]$ be irreducible. If $p(x)$ is not constant and not equal to an associate of $a(x)$, and we let $R=F[x]/(p(x))$, and identify F with its image in R, can $a(y)$ be reducible in $R[y]$?
 A: If you quotient by a reducible polynomial, you'll end up with zero divisors.
Consider $\mathbb{Q}[x]/(x^2-1)$. I'll refer to cosets "$f(x)+(x^2+1)$" just by their representatives "$f(x)$". 
Notice that $x+1,x-1$ are non-zero in this quotient, but $(x+1)(x-1)=x^2-1=0$ (mod $x^2-1$).
So we've got zero divisors.
If you want a better feel for what this ring "really" looks like. It's not hard to show that
$\mathbb{Q}[x]/(x^2-1) \cong \mathbb{Q}[x]/(x-1) \times \mathbb{Q}[x]/(x+1) \cong \mathbb{Q} \times \mathbb{Q}$. [Use the map $f(x) \mapsto (f(x)+(x-1),f(x)+(x+1))$ it's not hard to show this map is a homomorphism from $\mathbb{Q}[x]$ to $\mathbb{Q}[x]/(x-1) \times \mathbb{Q}[x]/(x+1)$ has kernel $(x^2-1)$ and using Chinese remaindering you can show it's onto. Then the result follows form the 1st isomorphism theorem.]
A: First, let's be clear:
If $R$ is a commutative ring with $1$, then an element $a\in R$ is irreducible if and only if $a$ is not a unit, $a$ is not a zero divisor, and whenever $a=bc$ in $R$, either $b$ is a unit (invertible) or $c$ is a unit. Added. Elements are not usually called "reducible", though. Elements that are not irreducible are "not irreducible". 
In the particular case of $R=F[x]$ with $F$ a field, this agrees with the "usual" notion of "irreducible polynomial", since the units are precisely the nonzero constant polynomials.
Now, if $F$ is a field, then $F[x]$ is a unique factorization domain (even more, a Euclidean domain). If $a(x)$ is irreducible, then it is a prime in $F[x]$. So for any $p(x)\in F[x]$ there are only three possibilities: either $a(x)$ divides $p(x)$, or $p(x)=0$, or else $a(x)$ and $p(x)$ are relatively prime, and in particular, there exist polynomials $r(x)$ and $s(x)$ such that
$$a(x)r(x) + p(x)s(x) = 1.$$
Now, suppose that $p(x)$ is a polynomial. The only way for $a(x)$ to be a multiple of $p(x)$ is for $a(x)$ to be of the form $a(x)=\lambda p(x)$ for some nonzero constant $\lambda\in F$.  Assume that $a(x)$ is not a multiple of $p(x)$.
If $p(x)\neq 0$ and $a(x)$ divides $p(x)$, then $p(x) = a(x)q(x)$ for some $q(x)\notin F$. We cannot have $p(x)$ dividing $q(x)$, so then $\overline{q(x)}$ is not zero in $F[x]/\langle p(x)\rangle$. Neither is $\overline{a(x)}$. Yet 
$$\overline{a(x)}\times\overline{q(x)} = \overline{a(x)q(x)} = \overline{p(x)} = 0.$$
So $\overline{a(x)}$ is not irreducible, because it is a zero divisor.
If $a(x)$ does not divide $p(x)$, then we can write $1 = a(x)r(x)+p(x)s(x)$ for some polynomials $r(x)$ and $s(x)$. But then in $F[x]/\langle p(x)\rangle$ we have:
$$1 = \overline{1} = \overline{a(x)r(x) + p(x)s(x)} = \overline{a(x)}\overline{r(x)} + \overline{p(x)}\overline{s(x)} = \overline{a(x)}\overline{r(x)},$$
so $\overline{a(x)}$ is not irreducible because it is a unit.
Finally, if $p(x)=0$, then $F[x]/\langle p(x)\rangle$ is essentially just $F[x]$ itself again, so $a(x)$ is irreducible in the image because it's just $a(x)$ in $F[x]$. 
So, to answer the question that was posed, if $p(x)\neq 0$ then $a(x)$ will not be irreducible in $F[x]/\langle p(x)\rangle$ no matter what. 

That said...
But your use of the word "reducible" suggests that you are actually wondering about something else: whether "as a polynomial" it can be written as a product of two nonconstant polynomials, or perhaps whether "it has a root" when considered in $F[x]/\langle p(x)\rangle$.
In other words: say $p(x)$ is a nonconstant polynomial in $F[x]$. Then $R=F[x]/\langle p(x)\rangle$ is a ring that contains a copy of $F$. We can then consider the polynomial $a(x)$ as having coefficients in $R$ and ask whether it is reducible or irreducible over $R$. However, using $x$ here is confusing because $x$ is already playing a role in the definition of $R$, so it's better to switch to a different letter in order to talk about "polynomials with coefficients in $R$". So let's use $y$ to denote polynomials with coefficients in $R$, and so we will be asking whether $a(y)$ is irreducible over $R$ or reducible over $R$.
If $p(x)$ is irreducible over $F$, then $F[x]/\langle p(x)\rangle$ is a field that contains $F$, and you can consider $a(y)$ as a polynomial in $(F[x]/\langle p(x)\rangle[y]=R[y]$. There, it may be that $a(y)$ is irreducible or that it is reducible. 
For an example where it is reducible, take $a(x) = x^4 + 1$ in $\mathbb{Q}[x]$, and $p(x) = x^2+1$, both irreducible. Since $\mathbb{Q}[x]/\langle p(x)\rangle = \mathbb{Q}(i)$, the rational complex numbers, in $\mathbb{Q}(i)[y]$ we have $a(y) = y^4+1 = (y^2+i)(y^2-i)$, reducible.
For an example where it is irreducible, take $a(x) = x^2 - 2$ and $p(x) = x^2-3$ in $\mathbb{Q}[x]$. Then $\mathbb{Q}[x]/\langle p(x)\rangle$ is just $\mathbb{Q}(\sqrt{3})$, and $a(y)$ has no roots in $\mathbb{Q}(\sqrt{3})$; being degree $2$, it is irreducible in $\mathbb{Q}(\sqrt{3})[y]$. 
If $p(x)$ is reducible, then you end up with a ring that is not a field and has zero divisors.  In any case, as a consequence of the Chinese Remainder Theorem, if we factor $p(x)$ into irreducibles,
$$p(x) = q_1(x)^{\alpha_1}\cdots q_m(x)^{\alpha_m},$$
with $q_i$ and $q_j$ coprime for $i\neq j$, $\alpha_i\gt 0$, then
$$R=F[x]/\langle p(x)\rangle \cong \frac{F[x]}{\langle q_1(x)^{\alpha_1}\rangle}\times\cdots\times \frac{F[x]}{\langle q_m(x)^{\alpha_m}\rangle},$$
and this $a(y)$ may or may not be irreducible over this ring. 
For an example where it is reducible, take the example above in which $a(y)$ was reducible, but replace $p(x)$ with $(x^4+1)(x^2+2)$ or some other reducible polynomial that has $x^4+1$ as a factor. Then $a(y)$ will have a root in the quotient, because it will have a root in one of the factors. Actually, since $F$ is embedded into the quotient "diagonally", you need two irreducible polynomials over which $a(x)$ was reducible, or a power of one over which $a(x)$ was reducible.  For an example where it is irreducible, do the same but now take the product of two polynomials over which $a(y)$ was irreducible. you can take $F=\mathbb{Q}$, $a(x) = x^2-2$, $p(x) = (x^2-3)(x^2-5)$. Then $F[x]/\langle p(x)\rangle \cong \mathbb{Q}(\sqrt{3})\times\mathbb{Q}(\sqrt{5})$, and that ring does not have a square root of $2$ either, so $a(y)$ is still irreducible. 
A: I will address what I believe you are trying to ask for in the first scenario.
Let $F = \mathbb{Q}$,  $a(x) = x^4-5$, $p(x) = x^2-5$.
Note that both $a(x)$ and $p(x)$ are both irreducible by Eisenstein criterion. But $F[x]/(p(x)) = F[\sqrt{5}] = F(\sqrt{5})$, and over the  polynomial ring $F(\sqrt{5})[x]$ we can factor $a(x) = (x^2-\sqrt{5})(x^2+\sqrt{5})$.
