Do the components of a natural transformation need to be part of the category? Given two functors, $F$ and $G$, between categories $\mathbf{C}$ and $\mathbf{D}$, a natural transformation $\eta$ associates a morphism $\eta_X$ for every $X$ in $\mathbf{C}$.  This morphism is between objects in $\mathbf{D}$. From what I read, that is (part of) the definition for a natural transformation.
My understanding of what is and is not a morphism is fuzzy.  Is there any requirement that $\eta_X$ be a morphism in $\mathbf{D}$ (i.e. $\eta_X \in \operatorname{hom}(\mathbf{D})$), or is it permitted to be any arbitrary mapping between two objects in $\mathbf{D}$?
As a concrete example, consider a very simple pair of categories, $\mathbf{C}$ and $\mathbf{D}$.

*

*Category $\mathbf{C}$ has two members, $A$ and $B$, and three morphisms: the two identity morphisms and $x$ where $x$ is from $A$ to $B$.

*Category $\mathbf{D}$ has four members, $A_1$, $A_2$, $B_1$ and $B_2$. It has six morphisms: the four identity morphisms, $x_1$ from $A_1$ to $B_1$ and $x_2$ from $A_2$ to $B_2$. There are obviously two functors from $\mathbf{C}$ to $\mathbf{D}$: $F$ maps $A$ to $A_1$, $B$ to $B_1$ and $x$ to $x_1$. $G$ maps $A$ to $A_2$, $B$ to $B_2$ and $x$ to $x_2$.

Is there a natural transformation between $F$ and $G$?  Clearly the transformation would have $\eta_A \colon A_1 \to A_2$ and $\eta_B \colon B_1 \to B_2$  However, neither $\eta_A$ nor $\eta_B$ are in $\operatorname{hom}(\mathbf{D})$. I can't tell if that's intended to be part of the definition or not.
If there is such a natural transformation between $F$ and $G$, is there a common name for the subcategory of natural transformations where $\eta_X$ is in $\mathbf{D}$ for all $X$ in $\mathbf{D}$?
 A: There are no “arbitrary mapping between two objects in $\mathbf{D}$”, and no morphisms between objects of $\mathbf{D}$ except for those in $\operatorname{hom}(\mathbf{D})$.
A “morphism in a category $\mathbf{D}$” is simply defined as “an element of $\operatorname{hom}(\mathbf{D})$”.
In other words:
we can only talk about morphisms in the context of an ambient category.

To define what a natural transformation is, we need the following data:

*

*two categories $\mathbf{C}$ and $\mathbf{D}$,

*two functors $F$ and $G$ from $\mathbf{C}$ to $\mathbf{D}$.

A natural transformation $η$ from $F$ to $G$ associates to every object $X$ of $\mathbf{C}$ a morphism $η_X$ from $F(X)$ to $G(X)$ in $\mathbf{D}$.
In other words: $η_X$ is by definition required to be an element of the set $\operatorname{Hom}_{\mathbf{D}}(F(X), G(X))$.
This means in particular that each $η_X$ is an element of $\operatorname{hom}(\mathbf{D})$.
To get to your specific example: you write that “neither $η_A$ nor $η_B$ are in $\operatorname{hom}(\mathbf{D})$”.
However, the problem is not that $η_A$ and $η_B$ exist outside of $\operatorname{hom}(\mathbf{D})$, but that they simply don’t exist.

There is, however, a way in which the idea of “morphisms outside of $\mathbf{D}$” can be made rigorous.
Suppose that $\mathbf{E}$ is a category that contains $\mathbf{D}$ as a subcategory.
Let $I$ be the inclusion functor from $\mathbf{D}$ to $\mathbf{E}$.
We may think about the functors $F$ and $G$ not only as functors from $\mathbf{C}$ to $\mathbf{D}$, but as functors from $\mathbf{C}$ to $\mathbf{E}$.
This means that we consider the two functors $I ∘ F$ to $I ∘ G$ instead of the original functors $F$ and $G$.
We can now study natural transformations between $I ∘ F$ and $I ∘ G$.
If $η$ is such a natural transformation from $I ∘ F$ and $I ∘ G$, then its components $η_X$ are morphisms in $\mathbf{E}$, i.e., they are elements of $\operatorname{hom}(\mathbf{E})$.
Now, $\operatorname{hom}(\mathbf{D})$ is a subclass of $\operatorname{hom}(\mathbf{E})$.
We can therefore ask whether every component $η_X$ is already an element of $\operatorname{hom}(\mathbf{D})$.
In other words:
it makes sense to ask whether all $η_X$ are already morphisms in $\mathbf{D}$.
If this is the case, then we can restrict $η$ to a natural transformation $\tilde{η}$ between the functors $F$ and $G$.
These natural transformations
$$
  η \colon I ∘ F \Rightarrow I ∘ G \,,
  \quad
  \tilde{η} \colon F \Rightarrow G
$$
have the same components, in the sense that $η_X = \tilde{η}_X$ for every object $X$ of $\mathbf{C}$.
But they are not the same natural transformation, because they are between different functors.
In this situation, there may be more transformations from $I ∘ F$ to $I ∘ G$ than there are natural transformations from $F$ to $G$.

To get back to your specific example:
We may define a new category $\mathbf{E}$ that has four objects and eight morphisms.
Its objects are $A_1$, $B_1$, $A_2$ and $B_2$.
Its morphisms are the four identity morphisms, a morphism $x_1$ from $A_1$ to $B_1$, a morphism $x_2$ from $A_2$ to $B_2$, a morphism $y$ from $A_1$ to $A_2$ and a morphism $z$ from $B_1$ to $B_2$.
Composition of morphisms is defined in the only way possible.
(This is the category corresponding to the partially ordered set with elements $A_1$, $B_1$, $A_2$, $B_2$ and partial order $≤$ given by $A_1 ≤ B_1$, $A_2 ≤ B_2$, $A_1 ≤ A_2$ and $B_1 ≤ B_2$.)
The category $\mathbf{D}$ is a subcategory of this new category $\mathbf{E}$.
Let $I$ be the inclusion functor from $\mathbf{D}$ to $\mathbf{E}$.
There exist no natural transformation from $F$ to $G$ because in the category $\mathbf{D}$ there exist no morphism from $A_1$ to $A_2$, as well as no morphism from $B_1$ to $B_2$.
But there exists a (unique) natural transformation $η$ from the functor $I ∘ F$ to the functor $I ∘ G$.
Its components are given by $η_A = y$ and $η_B = z$.
But $η$ is not a natural transformation from $F$ to $G$.
