How can this population extinct because of gender inequality? This population has the properties as follows:
(1) It is an isolated population, which means the individuals can only mate with others in this population.
(2) It is a monogamy population, which means each individual should only has one mate.
(3a) Each couple should give birth to N children. N is an integer. 
(3b)(alternative to 3a) Each couple gives birth to k children, and k follows a Poisson distribution: P(k)=$\frac{N^k}{k!}e^{-N}$. N can be a non-integer.
(4) The length of breeding cycle in this population is constant and consistent from individual to individual. All the individuals only give birth at their fixed breeding age. 
(5) There is a gene called DOOM. If one spouse in a couple carries the DOOM gene, they will give birth to male children at the probability of Pm. A couple without DOOM has the same probability to give birth to male children and female children.  
(6) If one spouse in a couple carries the DOOM gene, their children will carry DOOM gene.
(7) All the individuals can not distinguish those with DOOM from those without DOOM. 
(8) Initial conditions: The percentage of male DOOM carriers equals to female DOOM carriers, and is $q_0$. The percentage of male DOOM non-carriers equals to female DOOM non-carriers, and is $(1-2q_0)/2$. At the beginning, all the individuals are at their breeding age. 
[Question] In what condition given by N, Pm and $q_0$, this population will diminish gradually? 
 A: Maybe you can try to approach the problem this way:
Let $T_0 = m_0 + f_0$ be the initial population. And, in general, let $T_k$ be the population after the breeding cycle $k$; $m_k$ and $f_k$ the population of males and females of the $k$-th generation, respectively. Assuming ($3$a), that individuals have a life expectancy of $d$ breeding cycles, and that there's no breeding between different generations, we have:
$$T_k = N \min(m_{k-1}, f_{k-1}) + \sum_{j=\max(0,k-d)}^{k-1} m_j + f_j$$
Furthermore, if $P_d^k$ is the probability that an individual of the $k$-th generation has the DOOM gene, and $P_m$ is the probability that such an individual has male children, we have:
$m_k = N \left(\frac{1}{2}(1-P_d^k)+P_m P_d^k\right)\min(m_{k-1},f_{k-1})$
$f_k = N \left(\frac{1}{2}(1-P_d^k)+(1-P_m) P_d^k\right)\min(m_{k-1},f_{k-1})$
$P_d^k = P_d^{k-1}\left(2 - P_d^{k-1}\right)$
Note that $\{P_d^k\}_k$ is an increasing sequence. Now, set
$A_k(P_m) = N \left(\frac{1}{2}(1-P_d^k)+P_m P_d^k\right)$
$B_k(P_m) = N \left(\frac{1}{2}(1-P_d^k)+(1-P_m) P_d^k\right)$
From the recursive relations for $m_k$ and $f_k$ we have:
$m_k = A_k \min(A_{k-1}, B_{k-1}) \cdots \min(A_{1}, B_{1}) \min(m_0, f_0)$
$f_k = B_k \min(A_{k-1}, B_{k-1}) \cdots \min(A_{1}, B_{1}) \min(m_0, f_0)$
Now, you just have to study three cases:


*

*$P_m = \frac{1}{2}$:


In this case, $A_k = B_k$, so $\displaystyle m_k = f_k = 2 \min(m_0, f_0) \left(\frac{N}{2}\right)^k$, and $m_k$, $f_k$ converge iff $N = 0, 1, 2$. In that case, $\displaystyle \lim_k T_k = m_k + f_k = 0, d \min(m_0, f_0), 2d\min(m_0, f_0)$, respectively.


*

*$P_m > \frac{1}{2}$:

*$P_m < \frac{1}{2}$:
The last can be solved the same way.
A: I strongly suspect that basically either the doom gene spreads to the entire population very quickly or it goes extinct. Assuming the population size is large to begin with, and close to evenly split between males/females under the setup you described the doom gene will almost always spread to the whole population, so maybe let's just look at that case.
I will just model the number of breeding age couples of each generation $C_i$. From what I remember stochastic systems like this always go extinct in finite time. The expected value of the population size at time $i$ can still be an increasing function of $i$, so I guess you want to know under what circumstances that is true.
After a breeding event in case 3a we will always end up with $N C_i$ individuals, but the question is how many couples this corresponds to. Let's temporarily introduce the variable representing the number of males at time $i+1$, $M_{i+1}$. This follows the binomial distribution. And we have 
$C_{i+1} = \min(M_{i+1},N C_i - M_{i+1})$.
Let's assume $C_{i} \gg 1$ so we can use the normal approximation and seek an expression for the expectation of the ratio $\frac{C_{i+1}}{C_i}$.
$E[\frac{C_{i+1}}{C_i}] \approx \int_{-\infty}^{C_i N/2} \frac{m}{C_i} \frac{1}{\sqrt{2 \pi N C_i P_m (1-P_m)}} e^{-\frac{(m-N C_i P_m)^2}{2N C_i P_m (1-P_m)}}\mathrm{d}m + \int_{C_i N/2}^{\infty} \frac{N C_i  - m}{C_i} \frac{1}{\sqrt{2 \pi N C_i P_m (1-P_m)}} e^{-\frac{(m-N C_i P_m)^2}{2N C_i P_m (1-P_m)}}\mathrm{d}m$
I was able to do these in mathematica; and subsequently do a large $C_i$ expansion. Not surprisingly I found that to leading order 
$E[C_{i+1}/C_i] = N (1-P_m)$
So in order for the expected size of the population to grow you need the expected number of females born to each couple to be greater than $1$ (I assumed $P_m \ge 1/2$).
EDIT: I realized later that the integral asymptotic will come out very cleanly using the method of Laplace. I don't really want to put more time into this, but if you want a cleaner derivation apply this http://bit.ly/1bYXspG
A: I tried to make a recurrence sequence for this model as follows.
First, we make such definitions:
$N_{d,m}^{(k)}$ is number of male DOOM carriers in the $k$-th generation. 
$N_{d,f}^{(k)}$ is number of female DOOM carriers in the $k$-th generation. 
$N_{n,m}^{(k)}$ is number of male DOOM non-carriers in the $k$-th generation. 
$N_{n,f}^{(k)}$ is number of female DOOM non-carriers in the $k$-th generation. 
$T^{(k)}$ is number of couples in the $k$-th generation. 
$T_{d,d}^{(k)}$ is number of couples each consisting of 2 DOOM carriers in the $k$-th generation. 
$T_{d,n}^{(k)}$ is number of couples each consisting of 1 male DOOM carrier and 1 female DOOM non-carrier in the $k$-th generation. 
$T_{n,d}^{(k)}$ is number of couples each consisting of 1 male non-DOOM carrier and 1 female DOOM carrier in the $k$-th generation. 
$T_{n,n}^{(k)}$ is number of couples each consisting of 2 DOOM non-carriers in the $k$-th generation. 
Then we have:
$T^{(k)}=min((N_{d,m}^{(k)}+N_{n,m}^{(k)}) , (N_{d,f}^{(k)}+N_{n,f}^{(k)}))$
$T_{d,d}^{(k)}=T^{(k)}\frac{N_{d,m}^{(k)}}{N_{d,m}^{(k)}+N_{n,m}^{(k)}}\frac{N_{d,f}^{(k)}}{N_{d,f}^{(k)}+N_{n,f}^{(k)}}$
$T_{d,n}^{(k)}=T^{(k)}\frac{N_{d,m}^{(k)}}{N_{d,m}^{(k)}+N_{n,m}^{(k)}}\frac{N_{n,f}^{(k)}}{N_{d,f}^{(k)}+N_{n,f}^{(k)}}$
$T_{n,d}^{(k)}=T^{(k)}\frac{N_{n,m}^{(k)}}{N_{d,m}^{(k)}+N_{n,m}^{(k)}}\frac{N_{d,f}^{(k)}}{N_{d,f}^{(k)}+N_{n,f}^{(k)}}$
$T_{n,n}^{(k)}=T^{(k)}\frac{N_{n,m}^{(k)}}{N_{d,m}^{(k)}+N_{n,m}^{(k)}}\frac{N_{n,f}^{(k)}}{N_{d,f}^{(k)}+N_{n,f}^{(k)}}$
As assumed in the model, a couple with at least 1 DOOM carrier has the probability $P_m$ to give birth to boy. 
In the $k$-th generation, the number of couples with at least 1 DOOM carrier is $T_{d,d}^{(k)}+T_{d,n}^{(k)}+T_{n,d}^{(k)}$. 
Therefore, we have:
$N_{d,m}^{(k+1)}=P_m (T_{d,d}^{(k)}+T_{d,n}^{(k)}+T_{n,d}^{(k)}) N$
$N_{d,f}^{(k+1)}=(1-P_m) (T_{d,d}^{(k)}+T_{d,n}^{(k)}+T_{n,d}^{(k)}) N$
$N_{n,m}^{(k+1)}=N_{n,f}^{(k+1)}=0.5 T_{n,n}^{(k)} N$
Then we can obtain:
$\frac{(N_{d,m}^{(k+1)}+N_{d,f}^{(k+1)})}{(N_{n,m}^{(k+1)}+N_{n,f}^{(k+1)})}\frac{(N_{n,m}^{(k)}+N_{n,f}^{(k)})}{(N_{d,m}^{(k)}+N_{d,f}^{(k)})}=\frac{N_{d,m}^{(k)}N_{d,f}^{(k)}+N_{d,m}^{(k)}N_{n,f}^{(k)}+N_{n,m}^{(k)}N_{d,f}^{(k)}}{N_{n,m}^{(k)}N_{n,f}^{(k)}}\frac{(N_{n,m}^{(k)}+N_{n,f}^{(k)})}{(N_{d,m}^{(k)}+N_{d,f}^{(k)})}>1$
which means the ratio of DOOM carriers will always increase, indicating that the DOOM carriers will take up the entire population eventually. So we obtain
$\displaystyle \lim_{k \rightarrow\infty}N_{n,m}^{(k)}=0$ and $\displaystyle \lim_{k \rightarrow\infty}N_{n,f}^{(k)}=0$
Furthermore, we can have:
$\frac{T^{(k+1)}}{T^{(k)}}=\frac{min((N_{d,m}^{(k+1)}+N_{n,m}^{(k+1)}) , (N_{d,f}^{(k+1)}+N_{n,f}^{(k+1)}))}{T^{(k)}}= \frac{min((N_{d,m}^{(k+1)}+N_{n,m}^{(k+1)}) , (N_{d,f}^{(k+1)}+N_{n,f}^{(k+1)}))}{T^{(k)}}=\frac{min((\frac{P_{m}T^{(k)}(N_{d,m}^{(k)}N_{d,f}^{(k)}+N_{d,m}^{(k)}N_{n,f}^{(k)}+N_{n,m}^{(k)}N_{d,f}^{(k)})N}{(N_{d,m}^{(k)}+N_{n,m}^{(k)})(N_{d,f}^{(k)}+N_{n,f}^{(k)})}+N_{n,m}^{(k+1)}) , (\frac{(1-P_{m})T^{(k)}(N_{d,m}^{(k)}N_{d,f}^{(k)}+N_{d,m}^{(k)}N_{n,f}^{(k)}+N_{n,m}^{(k)}N_{d,f}^{(k)})N}{(N_{d,m}^{(k)}+N_{n,m}^{(k)})(N_{d,f}^{(k)}+N_{n,f}^{(k)})}+N_{n,f}^{(k+1)}))}{T^{(k)}}=min((\frac{P_{m}(N_{d,m}^{(k)}N_{d,f}^{(k)}+N_{d,m}^{(k)}N_{n,f}^{(k)}+N_{n,m}^{(k)}N_{d,f}^{(k)})N}{(N_{d,m}^{(k)}+N_{n,m}^{(k)})(N_{d,f}^{(k)}+N_{n,f}^{(k)})}+N_{n,m}^{(k+1)}) , (\frac{(1-P_{m})(N_{d,m}^{(k)}N_{d,f}^{(k)}+N_{d,m}^{(k)}N_{n,f}^{(k)}+N_{n,m}^{(k)}N_{d,f}^{(k)})N}{(N_{d,m}^{(k)}+N_{n,m}^{(k)})(N_{d,f}^{(k)}+N_{n,f}^{(k)})}+N_{n,f}^{(k+1)}))$
As $k \rightarrow\infty$, $N_{n,m}^{(k)}$ and $N_{n,f}^{(k)}$ $\rightarrow 0$, we have
$\displaystyle \lim_{k \rightarrow \infty}\frac{T^{(k+1)}}{T^{(k)}} =\lim_{k \rightarrow \infty} min(\frac{P_{m}(N_{d,m}^{(k)}N_{d,f}^{(k)}+N_{d,m}^{(k)}N_{n,f}^{(k)}+N_{n,m}^{(k)}N_{d,f}^{(k)})N}{(N_{d,m}^{(k)}+N_{n,m}^{(k)})(N_{d,f}^{(k)}+N_{n,f}^{(k)})},\frac{(1-P_{m})(N_{d,m}^{(k)}N_{d,f}^{(k)}+N_{d,m}^{(k)}N_{n,f}^{(k)}+N_{n,m}^{(k)}N_{d,f}^{(k)})N}{(N_{d,m}^{(k)}+N_{n,m}^{(k)})(N_{d,f}^{(k)}+N_{n,f}^{(k)})}) = min(P_{m}N,(1-P_{m})N)$
If N<2, $\frac{T^{(k+1)}}{T^{(k)}}<1$, which means the number of couples will decrease consistently in the long term, as a result, the population will extinct regardless of the value of $P_m$. 
When N>=2, 
If $(\frac{1}{N}-P_m)(1-\frac{1}{N}-P_m)<0$, $\displaystyle \lim_{k \rightarrow \infty}\frac{T^{(k+1)}}{T^{(k)}}>1$, and the population will increase. 
If $(\frac{1}{N}-P_m)(1-\frac{1}{N}-P_m)>0$, $\displaystyle \lim_{k \rightarrow \infty}\frac{T^{(k+1)}}{T^{(k)}}<1$, and the population will extinct. 
If $(\frac{1}{N}-P_m)(1-\frac{1}{N}-P_m)=0$, $\displaystyle \lim_{k \rightarrow \infty}\frac{T^{(k+1)}}{T^{(k)}}=1$, and the population will reach a stable size eventually. 
The initial conditions ($N_{d,m}^{(1)}$, $N_{d,f}^{(1)}$, $N_{n,m}^{(1)}$, $N_{n,f}^{(1)}$) will not affect the conclusions above if and only if these values are positive.
