This question was discussed on MathOverflow. See https://mathoverflow.net/questions/46684/fair-but-irregular-polyhedral-dice
The question on MathOverflow mentions a paper by Persi Diaconis and Joseph Keller, titled "Fair Dice", in which Diaconis and Keller argue that, by continuity, there are fair dice that are not simply fair by symmetry. Knowing a bit about Diaconis's philosophical stance on the interpretation of probability (I believe he is very strongly Bayesian), I was surprised by this.
But if you look at the paper, it states in the first paragraph of Section 3 that the shape of the die which would make it fair will depend upon the mechanical properties of the die and the table. This seems to be the essence of Matt Fayers's answer in the MathOverflow page, which states in part that "the only possible notion of a fair die is an isohedral one, because for any other die, it depends how you throw it."