I am assuming you have data from samples of two subsets of the population. With these samples you want to infer something about the population.
Is the mean of two subgroups of the population different?
Given sample sizes of 50 and 100*, you can conduct a two sample (unpaired) t-test. Almost every statistical package can do it; it is often used to compare the mean outcome in experiments between control and treatment group. In your example, it tells you if you can conclude with a given level of confidence that the share of "white" in the first subgroup is larger than in the second subgroup (or vice versa).
Since the data already contains confidence intervals computed by the census office, you can also use those. If the sample share of whites is, say, 50%, then the confidence intervals tell you the population share of this subgroup is in the intervals $[50- CI, 50+CI]$ for $\alpha$ samples drawn, where $\alpha$ is the confidence level (typically 95%). Therefore, if the confidence intervals of the two subgroups do not overlap, then the means are statistically different at least at confidence level $\alpha$. (But a slight overlap of confidence intervals need not imply that they are not statistically different; in that case, conduct the t-test.)
How to compare the median of two subgroups?
The t-test is not suitable to test this hypothesis, because it really tests equality of means. Most people would recommend the Wilcoxon rank sum test here, which is fine, but this is not strictly a test of the equality of median. It tests whether, say, incomes in one group tend to be higher than those in the other group (i.e., it compares ordinal ranks). Indeed, this is usually more interesting than the median. To specifically test equality of medians, you can use the median test.
*for considerably smaller sample sizes, you would have to check whether the data is normally distributed. For large samples, the mean is approximately normally distributed by the central limit theorem.