Do men or women have more brothers? Do men or women have more brothers?
I think women have more as no man can be his own brother. But how one can prove it rigorously?

I am going to suggest some reasonable background assumptions:


*

*There are a large number of individuals, of whom half are men and half are women.

*The individuals are partitioned into nonempty families.

*The distribution of the sizes of the families is deliberately not specified.

*However, in each family, the sex of each member is independent of the sexes of the other members.


I believe these assumptions are roughly correct for the world we actually live in.
Even in the absence of any information about point 3, what can one say about relative expectation of the random variables “Number of brothers of individual $I$, given that $I$ is female” and “Number of brothers of individual $I$, given that $I$ is male”?
And how can one directly refute the argument that claims that the second expectation should almost certainly be smaller than the first, based on the observation that in any single family, say with two girls and one boy, the girls have at least as many brothers as do the boys, and usually more.
 A: So many long answers! But really it's quite simple. 


*

*Mathematically, the expected number of brothers is the same for men and women.

*In real life, we can expect men to have slightly more brothers than women. 


Mathematically:
Assume, as the question puts it, that "in each family, the sex of each member is independent of the sexes of the other members". This is all we assume: we don't get to pick a particular set of families. (This is essential: If we were to choose the collection of families we consider, we can find collections where the men have more brothers, collections where the women have more brothers, or where the numbers are equal: we can get the answer to come out any way at all.) 
I'll write $p$ for the gender ratio, i.e. the proportion of all people who are men. In real life $p$ is close to 0.5, but this doesn't make any difference. In any random set of $n$ persons, the expected (average) number of men is $n\cdot p$.


*

*Take an arbitrary child $x$, and let $n$ be the number of children in $x$'s family. 

*Let $S(x)$ be the set of $x$'s siblings. Note that there are no gender-related restrictions on $S(x)$: It's just the set of children other than $x$.

*Obviously, the expected number of $x$'s brothers is the expected number of men in $S(x)$. 

*So what is the expected number of men in this set? Since $x$ has $n-1$ siblings, it's just $(n-1)\cdot p$, or approximately $(n-1)\div 2$, regardless of $x$'s gender. That's all there is to it.


Note that the gender of $x$ didn't figure in this calculation at all. If we were to choose an arbitrary boy or an arbitrary girl in step 1, the calculation would be exactly the same, since $S(x)$ is not dependent on $x$'s gender.
In real life: 
In reality, the gender distribution of children does depend on the parents a little bit (for biological reasons that are beyond the scope of math.se). I.e., the distribution of genders in families is not completely random. Suppose some couples cannot have boys, some might be unable to have girls, etc. In such a case, being male is evidence that your parents can have a boy, which (very) slightly raises the odds that you can have a brother. 
In other words: If the likelihood of having boys does depend on the family, men on average have more brothers, not fewer. (I am expressly putting aside the "family planning" scenario where people choose to have more children depending on the gender of the ones they have. If you allow this, anything could happen.)
A: Edit, 5/24/16: After some thought I don't particularly like this answer anymore; please take a look at my second answer below instead. 

Here's a simple version of the question. Suppose there is exactly one family which has $n$ children, of which $k$ are male with some probability $p_k$. When this happens, the men each have $k-1$ brothers, while the women have $k$ brothers. So it would seem that no matter what the probabilities $p_k$ are, the women will always have more brothers on average. 
However, this is not true, and the reason is that sometimes we might have $k = 0$ (no males) or $k = n$ (no females). In the first case the women have no brothers and the men don't exist, and in the second case the men have $n-1$ brothers and the women don't exist. In these cases it's unclear whether the question even makes sense.

Another simple version of the question, which avoids the previous problem and which I think is more realistic, is to suppose that there are two families with a total of $2n$ children between them, $n$ of which are male and $n$ of which are female, but now the children are split between the families in some random way. If there are $m$ male children in the first family and $f$ female children, then the average number of brothers a man has is
$$\frac{m(m-1) + (n-m)(n-m-1)}{n}$$
while the average number of brothers a woman has is
$$\frac{mf + (n-m)(n-f)}{n}.$$
The first quantity is big when $m$ is either big or small (in other words, when the distribution of male children is lopsided between the two families) while the second quantity is big when $m$ and $f$ are either both big or both small (in other words, when the distribution of male and female children are similar in the two families). If we suppose that "big" and "small" are disjoint and both occur with some probability $p \le \frac{1}{2}$ (say $p = \frac{1}{3}$ to be concrete), then the first case occurs with probability $2p$ (say $2 \frac{1}{3} = \frac{2}{3}$) while the second case occurs with probability $2p^2$ (say $2 \frac{1}{9} = \frac{2}{9}$). So heuristically, in this version of the question:

If it's easy for there to be many or few men in a family, men could have more brothers than women because it's easier for men to correlate with themselves than for women to correlate with men.

But you don't have to take my word for it: we can actually do the computation. Let me write $M$ for the random variable describing the number of men in the first family and $F$ for the random variable describing the number of women in the first family, and let's assume that they are 1) independent and 2) symmetric about $\frac{n}{2}$, so that in particular
$$\mathbb{E}(M) = \mathbb{E}(F) = \frac{n}{2}.$$ 
$M$ and $F$ are independent, so
$$\mathbb{E}(MF) = \mathbb{E}(M) \mathbb{E}(F) = \frac{n^2}{4}.$$
and similarly for $n-M$ and $n-F$. This is already enough to compute the expected number of brothers a woman has, which is (because $MF$ and $(n-M)(n-F)$ have the same distribution by assumption)
$$\frac{2}{n} \left( \mathbb{E}(MF) \right) = \frac{n}{2}.$$
In other words, the expected number of brothers a woman has is precisely the expected number of men in one family. This also follows from linearity of expectation.
Next we'll compute the expected number of brothers a man has. This is (again because $M(M-1)$ and $(n-M)(n-M-1)$ have the same distribution by assumption)
$$\frac{2}{n} \left( \mathbb{E}(M(M-1)) \right) = \frac{2}{n} \left( \mathbb{E}(M^2) - \frac{n}{2} \right) = \frac{2}{n} \left( \text{Var}(M) + \frac{n^2}{4} - \frac{n}{2} \right) = \frac{n}{2} - 1 + \frac{2 \text{Var}(M)}{n}$$
where we used $\text{Var}(M) = \mathbb{E}(M^2) - \mathbb{E}(M)^2$. As in Donkey_2009's answer, this computation reveals that the answer depends delicately on the variance of the number of men in one family (although be careful comparing these two answers: in Donkey_2009's answer he's choosing a random family to inspect while I'm choosing a random distribution of males and females among two families). More precisely,

Men have more brothers than women on average if and only if $\text{Var}(M)$ is strictly larger than $\frac{n}{2}$.

For example, if the men are distributed by independent coin flips, then we can compute that $\text{Var}(M) = \frac{n}{4}$, so in fact in this case women have more brothers than men (and this doesn't depend on the distribution of $F$ at all, as long as it's independent of $M$). Here the heuristic argument about bigness and smallness doesn't apply because the probability of $M$ deviating from its mean is quite small. 
But if, for example, $m$ is instead chosen uniformly at random among the possible values $0, 1, 2, \dots n$, then $\mathbb{E}(M^2) = \frac{n(2n+1)}{6}$, so $\text{Var}(M) = \frac{n(2n+1)}{6} - \frac{n^2}{4} = \frac{n^2}{12} + \frac{n}{6}$, which is quite a bit larger than in the previous case, and this gives about $\frac{2n}{3}$ expected brothers for men. 
One quibble you might have with the above model is that you might not think it's reasonable for $M$ and $F$ to be independent. On the one hand, some families just like having lots of children, so you might expect $M$ and $F$ to be correlated. On the other hand, some families don't like having lots of children, so you might expect $M$ and $F$ to be anticorrelated. Without the independence assumption the computation for women acquires an extra term, namely $\frac{2 \text{Cov}(M, F)}{n}$ (as in Donkey_2009's answer), and now the answer also depends on how large this is relative to $\text{Var}(M)$. 
Note that the argument in the OP that "no man can be his own brother" (basically, the $-1$ in $m(m-1)$) ought to imply, if it worked, that the difference between expected number of brothers for men and women is exactly $1$: this happens iff we are allowed to write $\mathbb{E}(M(M-1)) = \mathbb{E}(M) \mathbb{E}(M-1)$ iff $M$ is independent of itself iff it's constant iff $\text{Var}(M) = 0$. 

Edit: Perhaps the biggest objection you might have to the model above is that a given person's gender is not independent of the gender of their siblings; that is, as Greg Martin points out in the comments below, requirement 4 in the OP is not satisfied. This is easiest to see in the extreme case that $n = 1$: in that case we're only distributing one male and one female child, and so any siblings you have must have opposite gender from you. In general the fact that the number of male and female children is fixed here means that your siblings are slightly more likely to be a different gender from you. 
A more realistic model would be to both distribute the children randomly and to assign their genders randomly. Beyond that we should think more about how to model family sizes. 
A: I think I will argue that Cut the Knot is correct.  
The distribution of sizes of families is not specified.  Let's do some examples.  Suppose all families have size 1.  Then every boy has no brothers and every girl has no brothers.  (So we certainly cannot conclude girls have more brothers than boys independently of the distribution of family sizes.)  
Next example.  All families have size 2.  But random genders for the kids.  Then there are four types of families, all equally likely:
$$
B\qquad B\\
B\qquad G\\
G\qquad B\\
G\qquad G\
$$
I wrote B=boy, G=girl, in order of birth.  Now, if we choose a boy at random, how many brothers does he have?  There are four Bs in the list, two of them have 1 brother and two of them have no brothers.  So a boy chosen at random has no brother with probability $1/2$ and has one brother with probability $1/2$.  (Note: we chose a boy at random, not a family at random.) Now repeat, choosing a G at random.  We get: A girl chosen at random has no brother with probability $1/2$ and has one brother with probability $1/2$.  Again, it is false that a random girl has more brothers than a random boy.  
If you like, do it again for families of size 3.  A boy chosen at random has: no brother with probability $1/4$, one brother with probability $1/2$, and two brothers with probability $1/4$.  Same for a girl chosen at random.  
This works for any size families, as long as the sizes are fixed in advance, and the genders are random and independent.
A: To elaborate on MJD's edit clarifying the question, here are some assumptions sufficient for males and females to be "equal".


*

*no family planning based on gender. Within each family, the number of births is determined independently of the genders of children born to that family.

*probability 1/2 for every birth to give M or F as the gender 

*the number of families and distribution of family sizes in the total collection of families is determined independent of the genders of children that are/were born. 

*we compare $E[FM]$ to $E[M(M-1)]$ in each family, not number of brothers per male or female.  The latter are undefined if there is positive probability to have all the chilren be of one gender.
A: 
Do men or women have more brothers?

So many of the answers have probabilities in them but I see no reason for any argument to be made on the basis of probability.  We should simply count the number of men who have brothers and the number of women who have brothers, and compare those two numbers.
We can ignore all "only children" as they have no brothers.
Consider the people who have one sibling. Let's pair them off into sibling pairs, ordered by age. There are roughly equally many M-M, M-F, F-M and F-F pairs. Call the number of each n.  From the first group we have 2n males with brothers. From the second group we have n females with brothers. From the third group we have n females with brothers. From the fourth group we have 0 brothers. So we have a total of 2n males with brothers and 2n females with brothers. Neither side is winning in the two-sibling groups, so let's ignore them too.
Now consider the people who have two siblings. Again, group them into triples ordered by age. MMM, MMF, MFM, MFF, FMM, FMF, FFM, FFF. Suppose there are n of each group again. In the first we have 3n males with brothers. In the second, third and fifth we have 2n males with brothers. So that's a total of 9n males with brothers. Similarly we have 9n females with brothers.  So the 3-sibling groups are a wash as well. We can ignore them.
Does the pattern continue to 4, 5, 6 sibling groups?  Can you either find a group where there are an unequal number of females with brothers as males with brothers? Or can you find a pattern that lets you prove that they must always be the same?  
A: Consider a family with $n$ siblings of undetermined sex (each chosen independently at random). For each ordered pair of siblings, the probability that both are men is $1/4$. There are $n(n-1)$ ordered pairs, so the expectation for the sum of the number of brothers each man has is $n(n-1)/4$.
Similarly, the probability that the first sibling in the pair is a woman and the second sibling in the pair is a man is also $1/4$, so the expectation for the sum of the number of brothers each woman has is also $n(n-1)/4$.
We have shown that in each family of fixed size, the expected number of brothers of men is equal to the expected number of brothers of women. So it is also true for the population as a whole: the expected number of brothers of women is the same as the expected number of brothers of men.
A: Girls, in general, have more brothers than boys$^1$.
$^1$Given some assumptions and MATLAB simulations.
Here's my code: everything before the $---$ is the iterative procedure, and everything after is the function $\text{fplan}$ where most of the substance is.
n = 500; % number of boys and girls, each
f = 500; % number of families total

b_avg_count = 0;
g_avg_count = 0;

iter = 10000;

for i = 1:iter

    [b_avg, g_avg] = fplan(n, f);

    b_avg_count = b_avg_count + b_avg;
    g_avg_count = g_avg_count + g_avg;

end

b_avg_avg = b_avg_count/iter;
g_avg_avg = g_avg_count/iter;

---

function [b_avg, g_avg] = fplan(n, f)

% n = number of boys and girls, each
% f = number of families total
% n/f should be 1 to average ~2 kids per family

b = zeros(1, n);
g = ones(1, n);

c = [b g]; % generate all the children (boy = 0, girl = 1)

c = c(randperm(length(c))); % randomize ordering of children

f_sep = randperm(length(c)-1, f-1); % choose where families begin/end
f_sep = sort(f_sep);

f_test = [0 f_sep 2*n];
max_size = max(diff(f_test)); % determines largest family size

f_matrix = zeros(f, max_size);

b_amt = 0; % number of brothers had by the boys
g_amt = 0; % number of brothers had by the girls

for i = 1:f % iterate through all f families

    fam_num = c(1+f_test(i):f_test(i+1)); % generate i-th family
    fam_offset = 2*ones(1, max_size - length(fam_num));
    f_matrix(i, :) = [fam_num fam_offset];

    fam_bn = length(fam_num(fam_num == 0)); % number of boys in family
    fam_gn = length(fam_num(fam_num == 1)); % number of girls in family

    b_amt = b_amt + fam_bn*(fam_bn - 1); % each of B boys has B-1 brothers
    g_amt = g_amt + fam_bn*fam_gn; % each of G girls has B brothers

end

b_avg = b_amt/n; % average number of brothers had by the boys
g_avg = g_amt/n; % average number of brothers had by the girls

end

What things did I assume?


*

*There are an equal number of boys and girls.

*The amount of families is known, and no family has no children.

*You cannot be a brother to yourself.


(This first assumption is clearly very important. This case is very, very different from the case where we have $2n$ children and we assign each a gender with probability $\frac 12$.)
The gist of what I did was generate a vector with boys ($0$) and girls ($1$), randomize the order, and create families. How I created the families was to choose random distinct indices in my "children vector" and have families be all the children between these indices. This means that you can decide the average family size by choosing $\frac nf$, but the family sizes are "randomly" distributed.
Some cases ($200$ families with $200$ children) have deterministic outcomes (everyone is an only child), so choosing $n$ high, $f$ high, and $\frac nf \approx 1$ seems to be the most realistic way to approach the problem (average family size is $2$).
Given $1000$ children split between $500$ families, and iterating through this procedure $10000$ times, we get that:
$$\text{average brothers per male} = 0.9963$$
$$\text{average brothers per female} = 0.9984$$
This doesn't exactly answer the OP's question (in his/her view, the distribution of family sizes is entirely unknown), so take these results with a grain of salt. I think this model is realistic, however, and because of the high number of iterations and relative difference between the values, I'm heuristically convinced that girls have more brothers than boys.
I'm going to play around with the $n$ and $f$ parameters a bit, and if I find anything interesting, I'll try to whip up some graphs. The difference between the boy/girl values seems to increase as $f$ decreases, with deterministic behavior (obviously) at $f=1$.
A: There is some unspecified distribution of family sizes, according to which each person born may expect to have $s$ siblings, half of them brothers.
A: I've been accused of overcomplicating the issue, so here's a shorter and different answer. This mostly repeats things that have been said already, e.g. in zyx's answer. Consider any model where


*

*Children are male with probability $\frac{1}{2}$ and female with probability $\frac{1}{2}$,

*A given child's gender is independent of the gender of their siblings, and

*A given child's gender is also independent of the size of the family they're in.


With these assumptions, the expected number of brothers of any child is $\frac{F-1}{2}$ where $F$ is the expected size of a family (where we pick a random family by picking a random child). By linearity of expectation, the expected number of brothers of any man, as well as any woman, is also $\frac{F-1}{2}$. A simple example of a model satisfying all of these assumptions is a model where children are both distributed to a family uniformly and independently and also assigned a gender independently. Another example is a model where the size of each family is fixed, and genders are chosen independently. 
The model in my previous answer (dividing a fixed pool of children with fixed genders between two families) does not satisfy assumption 2. 
Interestingly enough, it can happen that there are no families with only male or only female children, meaning that in every family the women have more brothers than the men, and nevertheless it's still true that the expected number of brothers is the same for women and men. The reason is that when we compute the expectation for men, families with more male children are weighted more heavily. As Julian Rosen says in a comment on the OP, this is an example of Simpson's paradox. 
A: Assuming that there is an amount of men and women that does not equal $0$ in a family (not including the parents), let's say $3$ men and $3$ women, obviously a woman has $3$ brothers whereas one of her brothers only has $2$ brothers. Conversely, every man has more sisters than a woman does. To generalize it, if there are $n$ men and $m$ women for an arbitrary $n,m \in \Bbb N$, a woman has $n$ brothers while a man has $n-1$ brothers. Similarly, a man has $m$ sisters while one of his sisters has $m-1$ sisters.
A: Your assumption that females have more brothers is likely correct, but the reasoning you used is not valid.
"However, in each family, the sex of each member is independent of the sexes of the other members."
I don't believe that this is a valid assumption, in a society that assigns so much importance so much about gender.
Some in society might say "I will have children until I have a son, and then stop"
Possibilities are: 
M (1/2)
FM (1/4)
FFM (1/8)
FFFM (1/16)
And so on...
Average number of brothers:
Males: 0
Females: 1
Even if one were to go with a slightly different plan "I will have 2 children, and then have up to 2 more if I don't yet have a son" The odds are
MF (1/4)
MM (1/4)
FM (1/4)
FFM (1/8)
FFFM (1/16)
FFFF (1/16)
Average number of brothers:
Males: 4/16
Females: 1/4+1/4+1/8+1/16 = 11/16
Women have nearly 3 times as many brothers as men under these situations.
In scenarios where parents decide on number of children for reasons unrelated to gender, and where the breakdown is actually 50/50 (it isn't exactly, it depends on things like age, diet, stress etc), your assumption is incorrect - men cannot be their own brother, but females cannot be their own brother either. 
You cannot apply the assumption "families on average have equal numbers of males and females" to a particular family where you know the gender of one person in it, but have no information about the number of children or the gender of the others. You could try to apply the same logic to coin flips and this should highlight the issue more clearly. An average series of X coin flips will turn up equal numbers of heads and tails, but having seen the result of one of the coin flips, I now know that this series of X coin flips is more likely to have more heads, that is, the series of the other X-1 will still have even numbers of heads as tails.
A: There are many factors affecting this matter but the fundamental determinant of the answer is whether correlation within families is positive or negative, i.e. are parents having one male offspring more likely to have another male offspring, in relation to the overall population mean.
This could be due to a genetic disposition to having male children within certain families for example, or because of geographical variances in mortality rates.
If we look at countries where female children are aborted or killed after birth for example, having a son makes having another son slightly more likely since given that the family has a living son, it is more likely that that particular family is a practitioner of so-called "gendercide".  Thus brothers will have more brothers and fewer sisters.
The male predisposition for more dangerous occupations and activities is probably the most important cause of correlation.  For example in a region where there is war, men will generally be killed in greater proportion and therefore there will be a significant population of women with fewer brothers compared with the global population.  Since this region will not contain so many men with fewer brothers (since they too are dead), the impact will be to create more women with fewer brothers.
The contra argument would be that in regions where childbirth is dangerous there will be men with fewer sisters, since death in childbirth will again be co-rrelated within the family based on the simple fact that families are likely to be geographically local to each other. As a result, there will be fewer sisters around, and more men there having fewer sisters.
The most obvious way of balancing these arguments is to look at the mortality rate.  The fact that men die on average younger than women provides a strong argument that the effect will make more women have fewer brothers than vice versa.
We only therefore require the assumtions a) that correlation within a family is positive - e.g. because obviously family membership is correlated with geographical proximity, and b) that variations in mortality rate are correlated with geographical proximity and that will be sufficient to prove that women have fewer brothers than men do on the basis of men having a higher mortality rate than women.
So the answer is, men have more brothers than women do.
A: $\DeclareMathOperator{\ex}{\mathbb E}\DeclareMathOperator{\Var}{Var}\DeclareMathOperator{\Cov}{Cov}$If there are two giant families, one with all the women in it and one with all the men in it, then men will have more brothers on average.  So you cannot say anything in general, even assuming that there are the same number of men and women.  
So we will need to make some assumptions.  In order to find out what the correct assumptions are, I shall define some notation.  For convenience, I shall use probabilistic notation, but we are really just talking about counting.  
Let $M$ be the random variable corresponding to the number of men in a family chosen at random from the set of families, and let $W$ be the quantity corresponding to the number of women in a randomly chosen family.  Let $\mathcal M$ denote the total number of men, let $\mathcal W$ denote the total number of women and let $\mathcal F$ denote the total number of families.  

Important Note: The families themselves should be treated as constants.  They are not random samples drawn from some kind of
  distribution or anything like that.  When I use probabilistic
  notation, it is purely for the sake of convenience - I am interpreting the question combinatorially, and it so happens that probabilistic constructs such as sample variance do a good job of capturing certain combinatorial quantities that are relevant in this question.
If you like, we are working over the discrete measure space $(F,
\mathcal P(F), \mathbb P)$, where $F$ is the set of families and
  $\mathbb P(A)=|A|/|F|$ for any $A\subset F$.  $M$ is then the random
  variable defined by $M(f)=\textrm{number of men in $f$}$, while
  $W(f)=\textrm{number of women in $f$}$.  $\ex,\Var,\Cov$ will all take
  their usual meanings as population mean, population variance and population
  covariance.

We want to compute the average number of brothers that each man has.  To do this, we shall double-count the set $A$ of pairs $(m_1,m_2)$ such that $m_1$ and $m_2$ are brothers.  
The first way we count this set will be by family.  For a family $f$, let $m_f$ denote the number of men in family $f$.  Then we have
\begin{align}
|A|&=\sum_f m_f(m_f-1)\\
&=\mathcal F\ex(M(M-1))
\end{align}
since in each family $f$, we have $m_f$ choices for the first brother and $m_f-1$ choices for the second brother.  
The second way to count this set will be by man.  For a man $m$, denote by $b_m$ the number of brothers that $m$ has.  Then we have 
$$
|A|=\sum_m b_m
$$
Therefore:
$$
\sum_m b_m=\mathcal F\ex(M(M-1))
$$
Then:
\begin{align}
\textrm{Average number of brothers a man has}&=\sum_m b_m/\mathcal M\\
&=\frac{\mathcal F}{\mathcal M}\ex(M(M-1))\\
&=\frac{\ex(M(M-1))}{\ex M}
\end{align}
A similar argument gives us
$$
\textrm{Average number of brothers a woman has}=\frac{\ex(MW)}{\ex W}
$$
In this second case, if we write $w_f$ for the number of women in family $f$, then the number of pairs $(w,m)$ such that $w$ and $m$ are sister and brother is $m_fw_f$.  
We would like to show that this second quantity is bigger than the first.  Let's try and rewrite each quantity first.  
\begin{align}
\frac{\ex(M(M-1))}{\ex M}&=\frac1{\ex M}\left(\ex M^2-\ex M\right)\\
&=\frac1{\ex M}\left(\Var M+(\ex M)^2-\ex M\right)\\
&=\frac{\Var M}{\ex M} + \ex M - 1
\end{align}
while
\begin{align}
\frac{\ex(MW)}{\ex W}&=\frac{1}{\ex W}\left(\Cov(M,W)+\ex M\ex W\right)\\
&=\frac{\Cov(M,W)}{\ex W} + \ex M
\end{align}
Therefore, in order to ensure that the average number of brothers a woman has is greater than the average number of brothers a man has, we need to assume that:
$$
\frac{\Cov(M,W)}{\ex W}>\frac{\Var M}{\ex M}-1
$$
We can turn this into a condition saying that the number of men per family has to have small variance
$$
\Var M<\ex M+\frac{\mathcal M}{\mathcal W}\Cov(M, W)
$$
Is this a reasonable assumption to make?  Assuming that the number of men in a family is independent of the number of women in that family, we should expect $\Cov(M, W)$ to be small.  And if we assume that the total number of men is roughly equal to the to the number of women, then $\mathcal M/\mathcal W$ will be roughly equal to $1$.  So women have more brothers if and only if the variance in the number of men per family is less than $\ex M$.  
This is a surprising result, since there's no reason to suppose that the variance in the number of men per family should be less than $\ex M$.  In fact, numerical experiments indicate that $\Var M$ is quite often larger than $\ex M$, which means that in fact it will be men who have more brothers than women.  
So the answer is that it depends on how large the families are, but your intuition that women will have more brothers is not true in general, even under fairly strong assumptions.  If the number of men per family varies greatly from family to family, it is in fact men who have more brothers on average.  
This surprising result can easily be confirmed with numerical experiment.

What's the explanation?  Well, let's consider a situation in which the number of men per family varies greatly from family to family.  Suppose we assume also that the number of men per family is uncorrelated with the number of women per family.  
What this means is that there are going to be a significant number of families with lots of men and very few women, and a significant number of families with lots of women and very few men.  
Now, within any given family, we know that the women will have more brothers than the men.  But look at the overall contribution to the average: 


*

*The first type of family gives us lots of men with lots of brothers, and a small number of women with lots of brothers.

*The second type of family gives us lots of women with very few brothers, and a small number of men with very few brothers.  


So on average, the men will tend to have lots of brothers, and the women will tend to have very few brothers, as long as the variance in the number of men is large enough to counteract the extra brother that each woman has.  With large enough families, that one extra brother counts for less and less, and the variance effect takes over, giving you precisely the opposite effect from what you expected.  

Let's take one last look at the result.  We found that the important factor was that the variance in the number of men per family should not be too large.  What if this value were equal to zero?  That would mean that every family had the same number $a$ of men, so then it would be true that women have more brothers, since every man would have $a-1$ brothers and every woman would have $a$ brothers.  
The dependence on the covariance is interesting, too.  By the Cauchy-Schwarz inequality, we have $\Cov(M,W)\le\sqrt{\Var M\Var W}$, and if we assume that $\Var M$ and $\Var W$ are roughly the same, then we have $\Cov(M,W)\le\Var M$.  This extreme value occurs if $W=\lambda M$ for some positive constant $\lambda$.  In that case, a simple counting argument shows us that women will have an average of one more brother than men.  
A: In the case that we choose a person randomly and that the intra-family probabilities for each family are also $50\%:50\%$ there should be:
$p(boy)= p(girl)=50\%$
If the family size then is $n$ we would expect $n/2$ boys and girls respectively, or $n/2 - 1$ excluding the gender randomly picked. This gives:
$$(n/2-1) \cdot \frac{1}{2} + (n/2) \cdot \frac{1}{2} = \frac{1}{2}\cdot(n-1)$$
Which is we'd expect half of the remaining siblings $(n-1)$ to be girls.
However the conditional probabilities become $p(boy) = 0\%$ and $p(girl) = 100\%$ if it is given that the sibling picked was a girl. Then our calulation becomes:
$$(n/2-1) \cdot \frac{0}{1} + (n/2) \cdot \frac{1}{1} = \frac{1}{2}\cdot(n)$$
We can leave the case $p(boy)=100\%$ as an exercise we will land with $\frac{1}{1}(n/2-1)$
The fraction of the expected values $\frac{E[boys|girl]}{E[boys|boy]}$ will be $\frac{n-1}{n/2-1} = \frac{2n-2}{n-2}$. Since the numerator is larger for all non-negative integers $n$ (you can convince yourself by drawing a graph or using calculus) we can conclude we should expect more boys if we randomly picked a girl than if we randomly picked a boy.
A: If the (entirely unreasonable) assumptions 1-4 are taken as given and you want to examine only the properties of the fact that you have taken one "man" from your population, then we have to discard reality and the answer to this question is fundamentally down to the debate between Bayesian and conventional probability.  This depends on how "confident" you are in your rule that men and women are 50:50, and do you modify that estimator of the population mean in response to evidence, or do you hold it fast and adjust your knowledge of your sample in response to the sample of one taken.
If we are taking assumptions 1-4 then we have to ask how much of reality are we discarding?
If you discard it all then you are a computer with limited information and you probably have total confidence in the 50:50 rule.  When you take a man out under these circumstances, you know there to be one more woman in the remaining population and therefore women have more brothers than do men.
A family is a tiny sample of say 2-4 siblings from a population of several billion.  The fact that the brother is taken out of the population modifies the likelihood only very marginally.  But again, we are introducing the reality of a population of billions of people and a small family, and it is unclear how much of reality you want to eliminate.
If however you put less confidence in your 50:50 rule, the fact that you have taken a man from the population proves that a man exists and makes your best estimator of the percentage of men greater than 50%, but with assumptions 1-4 this leaves the answer unchanged.
Introduce a little bit more of reality and you know the 50:50 rule is the population mean deduced by observation and that the distribution has a binomial distribution very closely approximating the normal.  As such taking one man out leaves the distribution unchanged and therefore women and men have the same average number of brothers.
If you carry on introducing more of reality however, you will arrive at my other answer which is the correct answer - men have more brothers than women do.
A: This is really a comment but is too long (and I don't have the rep).
Monte Carlo confirms @Donkey_2009's answer; men have more brothers on average. 
I built a simple (and probably clumsy) excel spreadsheet with 2000 families, each with a random number (0...5) each of sons and daughters. Thus the family size (only counting kids) varies between 0 and 10. Within each family I calculated the number of brothers and sisters of boys and girls, then averaged them over all families. Typical results with this family size are that boys have 2.65 brothers and 2.45 sisters, similarly for girls. The spreadsheet is here. Typical formulae are:  
nBoys (column C) =INT(6*RAND())
nGirls (column D) =INT(6*RAND())
n_bros|B (column E) =IF(C2=0,0,C2*(C2-1)) [if no boys then zero, else Boys(Boys-1)]
n_bros|G (column F) =IF(D2=0,0,C2*D2) [if no girls then zero, else Boys*Girls]  
I'd be grateful if someone could check my calculations.
A: I'm going to provide an answer using basic probability rules and the Binomial distribution. Let $n$ be the number of children in a family.
Let $B$ be the number of boys among the $n$ children in a family - obviously, $B\leq n$, and I assume that
$$
B\sim \textrm{Binomial}(n,p)
$$
where $p$ is the probability of a male child. Now, I will use $M$ and $F$ to refer to male and female children, respectively.
$$
P(B=m)=\binom{n}{m}p^m(1-p)^{n-m}\\
P(M)=p\\
P(M|B=m)=\frac{m}n
$$
And so, it is easy enough to see that
$$
P(B=m|M)=\frac{m\binom{n}m p^m(1-p)^{n-m}}{np} = \binom{n-1}{m-1} p^{m-1}(1-p)^{n-m}
$$
And so, we have
$$
E(B-1| M) = \sum_{m=1}^n (m-1) P(B=m|M) = (n-1)p
$$
This is all as expected.
Similarly,
$$
P(B=m|F)=\frac{(n-m)\binom{n}m p^m(1-p)^{n-m}}{n(1-p)} = \binom{n-1}{m} p^{m}(1-p)^{n-1-m}
$$
so,
$$
E(B|F) = \sum_{m=0}^{n-1} m P(B=m|F) = (n-1)p
$$
Therefore, the number of brothers is the same for men and women, so long as independence is maintained. Based on the assumptions made in the question, we have $p=\frac12$, so both work out to $\frac{n-1}2$.
Note that this result is for a fixed $n$, and so the distribution of family size does not change the result.
Also note that the result changes if we change the distribution. For example, if $P(B=m)=\frac1{n+1}$ (for a uniform distribution of number of boys in a family of size $n$), then $E(B-1|M)=\frac{2(n-1)}3$ and $E(B|F)=\frac{n-1}3$, so men will have twice as many brothers as women will.
In reality, it is likely that a small bias in favour of men having more brothers will occur as the probability of a male child in one family is likely fixed but specific to that family, and so children of the same gender are slightly more likely than children of different genders.
A: Correct me if I'm wrong but this is just a more complicated version of the Boy or Girl Paradox. https://en.wikipedia.org/wiki/Boy_or_Girl_paradox
For sake of simplifying the problem, let's say each family only has two children. Then, if you randomly take a family and ask if there's a girl, then what's the chance that she has a brother? Probability will obviously be 2/3.
On the other hand, say a family pops out a girl, then they pop out another child. Probability is obviously 1/2. This is just another paradox which shows that the exact random event generation matters.
So, to address OP's question, OP needs to answer which scenario is occurring.
