Are there novels (or other kinds of books) that include substantial references to topics and ideas closely related to mathematics (even if there are no explicit references to theorems, proofs, ...)?
I am surprised that no one has mentioned Jorge Luis Borges well regarded novels.
In particular The Library of Babel and The Book of Sand, both of which deal with infinity and set theory amongst other themes such as the related philosophical themes of Kant and Hume.
Both of these books have been cited as influential by contemporary novelists such as Umberto Eco and philosophers such as Quine and Dennett.
Hiroshi Yuki's Math Girls (which is available in English translation from the original Japanese) is mostly mathematics, with a thin story about the narrator's relationship with two girls with whom he studies mathematics.
Much better from a literary standpoint is Yoko Ogawa's The Housekeeper and the Professor (also available in translation from the original Japanese). The housekeeper cares for a former mathematician who has lost his ability to form new memories in a car accident. However, he does remember his mathematics. He also likes children. When the professor meets the housekeeper's son, he establishes a relationship with the boy (whose name he can not remember) by teaching him mathematics.
The Oulipo is a group of novelists, poets and mathematicians, trying to make their writing more original by imposing combinatorial constraints on the text, or playing various combinatorial games with it.
One of the best known examples is Raymond Queneau's Cent mille milliards de poèmes (100,000,000,000,000 poems, this large set of poems can be described succinctly as a cartesian product of smaller sets).
Another more contrived example uses the Fibonacci sequence and Zeckendorf's theorem to constrain the semantic dependences (and the rhymes, as well) of the verses of a poem (Anthologie de l'Oulipo, M. Benabou and P. Fournel editors, Gallimard, 2009, page 158).
Change ringing was somewhat similar, but the combinatorial constraints were applied to music.
The play "Arcadia" by Tom Stoppard comes to mind. http://en.wikipedia.org/wiki/Arcadia_%28play%29
references to Iteration and Chaos Theory ; Nonlinear Dynamics if you prefer. It is thought that Tom Stoppard based the character of Thomasina Coverly on Ada Lovelace.
From the Synopsis on Wikipedia: In 1809, Thomasina Coverly, the daughter of the house, is a precocious teenager with ideas about mathematics, nature and physics well ahead of her time. She studies with her tutor Septimus Hodge, a friend of Lord Byron (an unseen guest in the house). In the present, writer Hannah Jarvis and literature professor Bernard Nightingale converge on the house: she is investigating a hermit who once lived on the grounds, he is researching a mysterious chapter in the life of Byron. As their studies unfold -- with the help of Valentine Coverly, a post-graduate student in mathematical biology -- the truth about what happened in Thomasina's time is gradually revealed.
The play's set features a large table, used by the characters in both past and present. Props are not removed when the play switches time period; books, coffee mugs, quill pens, portfolios, and laptop computers appear together, blurring past and present. An ancient but still living tortoise also appears in every scene, symbolising long-suffering endurance and the continuity of existence.
The Curious Incident of the Dog in the Night-Time is told from the viewpoint of a child with Asperger's who loves math. It has many instances of the narrator explaining cool math concepts to the reader, and trying to use math to make sense of his life. Also, it is very touching.
I understand that much of Greg Egan's work could fit here, but I have only read his "Permutation City". A large part of the plot of this book involves implementing an ever growing computer architecture inside a 4 dimensional version of Conway's "Game of Life". Fantastic novel with a lot of real mathematical concepts. If some dust in alpha centari happens to be isomorphic to your neural architexture, is that dust cloud you?
I also seem to recall a pretty cool exposition, in story form, of a link between the mathematics of elliptical orbits and quantum spins.
Greg Egan has also done some "real" mathematical work, I believe coauthoring a paper with John Baez.
Against the day by Thomas Pynchon is a prime example (with multiple generally meaningful references to quaternion algebras, topology, zeta functions and the like). Michael Harris commented entertainingly on math in Pynchon's work here.
This is probably a very subjective answer to your non-mathematician criteria, but I have enjoyed books by authors such as William Faulkner and Cormac McCarthy whose narratives often make use of logical implications and negations. Plays by writers such as Arthur Miller, Henry James, and Eugene O'Neill, just to name a few, are also embedded with a lot of examples of deductive reasoning as is often seen in Mathematics. But then again, I spent my undergraduate years reading and writing a lot since I was a Mathematics major and an English minor.
"Lovesong of the electric bear", about Alan Turing, including substantial discussion of his work.
Also, this might be of interest: A big list of math fiction!
The Name of the Rose by Umberto Eco contains substantial discussions of mathematics as it may have been conceived by an intellectual of the early 14th century and math ends up playing a crucial role in the plot. However, as it contains substantial discussions of every intellectual endeavors as they may have been conceived by such a character and as they all play an important role at some point, math is nothing special in that regard.
Alice's Adventures in Wonderland, in particular: http://en.wikipedia.org/wiki/Alice's_Adventures_in_Wonderland#Symbolism
Mostly implicitly referencing, as far as I recall.
The science-fiction novel Rocheworld by Robert Forward had "happy-go-lucky aliens [called Flouwen] that spend their days surfing waves and working on difficult mathematical problems." Also, "Flouwen possess mathematical abilities far exceeding our own."
They tend to treat one another equally, though mathematics proficiency appears to confer a heightened social status. Mathematics is one of the few subjects in which Flouwen show interest and concern. While younger Flouwen seem to have large amounts of free time, their elders spend long periods of time in rock form, contemplating and solving mathematical problems. As a result, the older Flouwen often hold higher social status as a result of their perceived higher knowledge in mathematics.
The novel presents some of the mathematical problems that Flouwen would give to their young. The ones I remember involve cardinalities (infinite sets of different sizes).