About the formulas to determine the integral ideal with specific norm (1) For $\mathbb{Q}(\sqrt{10})$, the ring of integers is $\mathbb{Z}[\sqrt{10}]$, and to determine the integral ideal with norm $3$, we can use the formula below:
By $X^2 − 10\equiv (X + 1)(X − 1)\text{ mod 3}$, then we get the conjugate ideals with norm 3: $(3, \sqrt{10}+1), (3, \sqrt{10}-1)$.
(2) For $\mathbb{Q}(\sqrt{17})$  the ring of integers is $\mathbb{Z}[\alpha]$, where $\alpha=\frac{1+\sqrt{17}}{2}$ and to determine the integral ideal with norm $2$, we can use the formula below:
$0=(\alpha-1/2)^2-17/4=\alpha^2-\alpha-4$ $X^2 − X − 4 ≡ X(X + 1) \text{ mod 2}$ then we get the conjugate ideals with norm 2: $(2,\alpha),(2,\alpha+1)$.
I am confused with the reason of these two formulas, where are these formulas from. Actually, I can't find any reference to illustrate these formulas. Can anybody explain these two formulas? Thank you!
 A: This answer, despite its length, is not a complete answer either, but I think it'll give you some insights. Your question feels like another one I've answered before, although I think the earlier question was more concerned with numbers than with ideals.
Regardless, I feel that the excessive early emphasis on defining $\alpha$ (sometimes $\theta$ or $\omega$, but I think the latter should be reserved for a specific ring) only serves to confuse students.
However, I am not certain the formulas you were given are correct, but I could be wrong, as I've approached this topic from a different, far less systematic path than yours.
Given a squarefree integer $d$, the norm function is defined for any number in $\mathbb Q(\sqrt d)$, whether or not that number is also in the ring of algebraic integers. Namely, if $a$ and $b$ are rational real numbers, then $N(a + b \sqrt d) = a^2 - db^2$. And if $a + b \sqrt d$ is an algebraic integer, then its norm is an ordinary integer.
For example, consider the number $$\frac{3 + \sqrt{10}}{2}.$$ Then $$N\left(\frac{3 + \sqrt{10}}{2}\right) = \left(\frac{3}{2}\right)^2 - 10\left(\frac{1}{2}\right)^2 = \frac{9}{4} - \frac{10}{4} = -\frac{1}{4},$$ which is not an integer.
In fact, for a number $a + b\sqrt{10}$ to be an algebraic integer, $a$ and $b$ must both be ordinary integers. From there it is easy to see that $a - 10b = \pm 3$ has no solutions, hence no number in this domain can have a norm of 3, and so the ideal with that norm can't be a principal ideal.
The smallest positive integer multiple of 3 that is a possible norm in this domain is 6, but 9 works, too, giving us the ideals $\langle 3, 1 - \sqrt{10} \rangle$ and $\langle 3, 1 + \sqrt{10} \rangle$.
Now let's go to $\mathbb Q(\sqrt{17})$, and consider the number $$\frac{3 + \sqrt{17}}{2}.$$ Then $$N\left(\frac{3 + \sqrt{17}}{2}\right) = \left(\frac{3}{2}\right)^2 - 17\left(\frac{1}{2}\right)^2 = \frac{9}{4} - \frac{17}{4} = -\frac{8}{4} = -2,$$ which is an integer.
Since the fundamental unit of this domain is of norm 1 rather than norm $-1$, a number with a norm of 2 is impossible. (Oops, silly mistake, there's $29 + 7\sqrt{17}$ divided by 2, and infinitely many others. I stand by everything else.)
But with ideals, we don't have to worry about signs as much, and so $$\langle 2 \rangle = \left\langle \frac{3 - \sqrt{17}}{2} \right\rangle \left\langle \frac{3 + \sqrt{17}}{2} \right\rangle,$$ a product of principal ideals.
Of course if you want to interpret those in terms of $$\alpha = \frac{1 + \sqrt{17}}{2},$$ you can, like so:
$$\frac{3}{2} - \frac{\sqrt{17}}{2} = \frac{4}{2} - \left(\frac{1}{2} + \frac{\sqrt{17}}{2}\right) = 2 - \alpha$$ and $$\frac{3}{2} + \frac{\sqrt{17}}{2} = \frac{2}{2} + \left(\frac{1}{2} + \frac{\sqrt{17}}{2}\right) = 1 + \alpha.$$
You should recognize the latter number as a co-generator of one of the ideals you found: $\langle 2, 1 + \alpha \rangle$. But since $(2 - \alpha)(1 + \alpha) = -2$, the ideal $\langle 2 \rangle$ is contained in the ideal $\langle 1 + \alpha \rangle$, and therefore 2 is a redundant generator in $\langle 2, 1 + \alpha \rangle$.
As for the ideal $\langle 2, \alpha \rangle$, that can also be boiled down to a principal ideal, namely $\langle \gcd(2, \alpha) \rangle$, but I don't think that's a prime ideal, since, for starters, $N(\alpha) = -4$.
A: Contrary to the algorithmic answer of @Will Jagy, mine will be theoretical, based on the decomposition of primes in a number ring. Let us first fix definitions and notations. For a number field $K$ with ring of integers $A$, the (numerical) norm of an ideal $I$ of $A$ is the integer $N(I)$ = order of $A/I$. But one can also consider the principal ideal $(N(I))$ generated by $N(I)$ in $A$ and look at its (unique) decomposition into powers of distinct prime ideals of $A$, say $N(I)=Q_1^{e_1} ... Q_r ^{e_r }$ (*), where $e_j$ is the ramification index of $Q_j$. In the case $K/\mathbf Q$ is Galois, all the $Q_j$ ' s are mutually conjugate under the action of $Gal(K/\mathbf Q)$ and all the $e_j$ 's are the same. Note that the situation carries over almost textually to the relative case, i.e. when replacing $\mathbf Q$ by a number field $k$. 
See e.g. The product of all the conjugates of an ideal is a principal ideal generated by the norm. 
Thus your problem of ideals of a quadratic field $K=\mathbf Q (\sqrt d)$ having prime norm $p$ boils down to the split decomposition $(p) = Q_1.Q_2$ (this happens iff $p=2$ and $d \equiv 1$ mod $8$ , or $p$ odd and $(\frac dp)=1$). In general, finding the decomposition (*) above is not an easy task. But there is a  theorem of Dedekind which works "almost" all the time. You can find it e.g. in Marcus' book "Number Fields", chapter 3, thm. 27. The necessary hypothesis is : if $K=\mathbf Q(\alpha)$, $p$ does not divide the index $[A:\mathbf Z[\alpha]]$. Then, denoting by $g$ the monic minimal polynomial of $\alpha$, $\bar g = \bar g^{e_1} ... \bar g^{e_r}$ the factorization of the image of $g$ in $\mathbf F_p [X] $, one has $Q_j = (p, g_j (\alpha))$. This exactly what is done in your first example.
At first sight, Dedekind's theorem does not apply to your second example. Fortunately, in the particular case of a quadratic field $\mathbf Q (\sqrt d)$, the theorem can be sharpened to cover all cases except when $p=2$ and $d \equiv 1$ mod $4$ , and in this exceptional case, the result can be obtained by taking $\alpha=(1+\sqrt d)/2$  (op. cit., comments at the end of chap. 3, p. 82)  ./.
A: framework at http://math.blogoverflow.com/2014/08/23/binary-quadratic-forms-over-the-rational-integers-and-class-numbers-of-quadratic-%EF%AC%81elds/
In case of interest: the classes for $\Delta = 40$ are represented by


*

*1           6          -1   cycle length             2

*2           4          -3   cycle length             6


which you might prefer as classes $x^2 - 10 y^2$ and $2 x^2 - 5 y^2.$ The latter does integrally represent $3,$ the full Lagrange Gauss cycle is
0  form   2 4 -3   delta  -1     ambiguous  
1  form   -3 2 3   delta  1
2  form   3 4 -2   delta  -2
3  form   -2 4 3   delta  1     ambiguous            -1 composed with form zero  
4  form   3 2 -3   delta  -1
5  form   -3 4 2   delta  2
6  form   2 4 -3

The mapping from forms to ideals is
$$ \langle A,B,C \rangle  \mapsto  \left( A, \frac{B + \sqrt \Delta}{2} \right),  $$ while $\sqrt {40} = 2 \sqrt {10}.$
$ \langle 3,4,-2 \rangle $  and $ \langle 3,2,-3 \rangle $ are $SL_2 \mathbb Z$ equivalent,  while the first is equivalent to $ \langle 3,-2,-3 \rangle $ by a single step. We get
$$ \langle 3,-2,-3 \rangle  \mapsto  \left( 3, \sqrt {10} - 1 \right),  $$
$$ \langle 3,2,-3 \rangle  \mapsto  \left( 3, \sqrt {10} + 1 \right).  $$
For $17,$   just one class


*

*1           3          -2   cycle length             6


with cycle
0  form   1 3 -2   delta  -1     ambiguous  
1  form   -2 1 2   delta  1
2  form   2 3 -1   delta  -3
3  form   -1 3 2   delta  1     ambiguous            -1 composed with form zero  
4  form   2 1 -2   delta  -1
5  form   -2 3 1   delta  3
6  form   1 3 -2

$$ \langle 2,1,-2 \rangle  \mapsto  \left( 2, \frac{1 + \sqrt {17}}{2} \right),  $$
$$ \langle 2,3,-1 \rangle  \mapsto  \left( 2, \frac{3 + \sqrt {17}}{2} \right),  $$
