Number of ways of build a binary matrix with constraints We have an $N\times N$ matrix U whose elements can only be $0$ or $1$. I would like to count the multiplicity of the following matrix:


*

*$E$ non-zero entries 

*$\sum_i U_{i j} = a_j$ i.e. fixed sum along each row

*$\sum_j U_{i j} = b_j$ i.e. fixed sum along each column
At first I thought to map the problem as a "distinguishable balls into distinguishable boxes" problem but with no results. Now I am thinking that maybe a solution can be reached using the Stirling numbers of the first kind since the problem can be traced to analysing the permutations of the sequence $(u^1_1 u^1_2 ... u^1_{a_1})(u^2_1 u^2_2 ... u^2_{a_2})...(u^N_1 u^N_2 ... u^N_{a_N})$ by using the interpretation that $u^1_i=5$ it means that $U_{1 5} = 1$.
Any idea? 
 A: From a008300 in oeis, it seems already difficult when $a_j$ and $b_i$ are all the same number.
If I assume the rowsums are correct independently of the column sums,
I get a ballpark figure
$$\frac{f(a_1,...,a_N)f(b_1,...,b_N)}{N^2\choose E}$$
This won't be correct, for example if $\vec a=(3,0,0)$ and $\vec b=(2,1,0)$ there are no common solutions.  And when $\vec a=(2,2,2,2)=\vec b$, the estimate is 130.5 instead of the true answer of 90.
EDIT:
$$f(a_1,...a_N)={N\choose a_1}{N\choose a_2}...{N\choose a_N}$$
When all $a_j$ and $b_i$ are 2, the 1s in the matrix form loops, as you go from any 1 to the other 1 in the same row, then the other 1 in the same column, then the other 1 in the same row, and so on.   
The number of ways to form a loop that includes the 1s in the first row, and involves $k$ rows and columns, is $\frac1{2N}(N!)^2/((N-k)!)^2$.  
If $g(N,2)$ is the number of ways for an $N\times N$ matrix when all $a_j=2$ and all $b_i=2$, then
$$g(N,2)=\frac1{2N}\sum_{k=2}^N\left(\frac{N!}{(N-k)!}\right)^2g(N-k,2)$$
with initial conditions $g(0,2)=1,g(1,2)=0$
