Margin of error in relation to population size. As I understand it, the population size does not factor in to margin of error on polls. In a theoretical world this makes sense, but in reality there are often data points which are not accessible making your "random" sample less random which is absolutely fundamental to the equation.
Suppose for example the total population size was 1 million and you want to provide a poll with no more than a 2.5% margin of error so you have to randomly sample 1600 people right? But what if 100,000 of those people are out on vacation and you have no way to include them in the pool from which you randomly select. How might you adjust the math to say what the margin of error really is in this scenario? 
 A: Sampling bias. If the people who cannot be reached or who refuse to
participate have different opinions than those who are
truly available, then there is no mathematical solution.
Based on past data and experience different pollsters have various opinions how to compensate for the inability to sample from the entire population of interest. 
Sample size. If you are strictly interested in the effect of decreased population size, then you are correct that population size does not ordinarily play a role in
the reporting of polling results. 
For example, consider two polls--one in a population
of 1 million and another in a population of 50 million.
Suppose the proportion with a particular view of 
a social issue is 55% in both populations.
Then responses for about $n = 2500$ randomly chosen
subjects are required in both populations to get
a margin of sampling error of about $\pm 2\%.$
Sampling with and without replacement. This result is based on the ideal of sampling without
replacement from each population. if the 1340 out of 2500 randomly chosen respondents take that position
then the point estimate of the probability is
$\hat p =1381/2500 =  0.5524,$ the 95% margin of
sampling error is $1.96\sqrt{\frac{\hat p(1-\hat p)}{n}}
= 0.0194,$ and a 95% confidence interval for the
true proportion in favor is about $55.2\% \pm 1.9%.$
Of course, in practice, it is difficult to enforce
'sampling without replacement': it is always possible (even if unlikely) that several of the 2500 respondents were inadvertently contacted twice. Essentially,
this is about the difference between a binomial model (which allows for repeat contacts)
and a hypergeometric model (which does not). 
Finite population correction. There
is a 'finite population correction' to adjust the
margin of error for possible repeat contacts, but it
is too small to be worth considering as long as the
population size $N$ is 10 or 20 times the sample size $n.$ Public opinion polls are seldom conducted in populations as small as $20(2500) = 50\,000,$ so this correction is
rarely necessary. (Who would pay for that?)
Similarity of binomial and hypergeometric models for large populations. By way of illustration, the two distributions plotted
below are for a binomial model with $n = 2500$ and $p = .55$ (blue bars) and a hypergeometric model with $n = 2500$ from a population with 55000 in favor and 45000 opposed (centers of red circles). At the resolution of the graph,
it is difficult to distinguish between the two distributions.

