Short answer: a ton.
A paper that I'm unable to access (maybe if you are at an educational institution you can access it through the library) can be found here: Anti-counterfeit Image Analysis Methods
The link just goes to the abstract which includes the paragraph, "Machine-readable travel documents are well-suited examples introducing the second group of methods which are related to cryptography. Used in passports for data authentication and identification (of people), cryptography provides some powerful tools. Opto-digital processing allows some efficient implementations described in the papers and promising applications."
This sounds like it might offer a lot of information and other good references.
A really in depth paper on multifactor optical identification systems can be found here. (Titled Encryption and validation of multiple signals for optical identification systems).
I'm not sure it specifically addresses passports, but this is the sort of mathematics that would go into verifying a passport based on optical analysis. My understanding is that there are many hidden/invisible optical features actually embedded into the passport and optical readers that are used to verify the passport's authenticity take a random sampling of different regions of the passport and compare the properties of that sample to the expected properties from a genuine document. I'm far from an expert in the field but the above link should point you in the right direction.
I also think a google search on Stereology and anti-counterfeiting might turn up some interesting hits. Various new methods in stereology are certainly making their way into anti-counterfeiting schemes for currencies and documents.
Hopefully there are some people out there with far more expertise than I have who can give some actual insight to these methods. I find it a fascinating area of applied mathematics.