Statistics. How are standard error and confidence intervals useful without knowing population size?

I understand standard error and confidence intervals as formulas, but not as concepts. Can you help me understand them better?

A smaller standard deviation (smaller spread of your data) and a larger sample size both give you a smaller standard error. That in turn gives you a narrower confidence interval.

In layman's terms: As your data points move closer to the sample mean; and as your sample size (n) gets closer to your population size (N), you can be more confident that your sample statistic matches your population parameter.

But how do you calculate your sample confidence is your don't know your population size? An example I threw together in Excel:

You want to know how the median sick days workers in your town take each year. You survey companies and get responses for 36 workers. The mean for the 36 workers is 14.64 days (I'm rounding). The standard deviation is 9.30. That gives you a standard error of 1.55 and a 95% confidence interval of +-3.15.

You conclude, "I'm 95% sure that workers in our town take between 11 and 17 sick days per year."

But how do you know that estimate is even close? If your little town has only 100 workers, then a survey of 36 is pretty accurate. If you have 100,000 workers in your town, your sample is probably way off. The formulas for standard error and confidence interval (as well as standard deviation) don't have N in their calculations.

In many cases, you don't even know N (number of frogs in a national park; amount of drugs smuggled through an area; tons of ore in a mine). So how do you calculate (percent of frogs with a disease; percentage of drugs stopped; quantity of ore per ton of rock) without knowing N? Is a thousand frogs sufficient? Is a hundred bricks of pot a good job? If we extract 16 tons, what do we get?

Corollary to this: If we know N, can we use ti change our statistics for n?

This is a repeat of How is it that the required sample size for a specified error and confidence is not dependent on population size?, but I don't grasp the concept of infinite populations.