I couldn't decide whether this question is more appropriate to post here or at the philosophy SE, but I thought I'd give people with a mathematical perspective the opportunity to help me decide. I'm happy to migrate the question if people think it necessary.
Anyway, the question is about a famous quote in von Neumann's The Computer and the Brain (published posthumously), that goes:
When we talk mathematics, we may be discussing a secondary language, built on the primary language truly used by the central nervous system.
Thus logics and mathematics in the central nervous system, when viewed as languages, must structurally be essentially different from those languages to which our common experience refers.
I've read the relevant passages a few times, and far as I can see he is trying to convey a statement along the lines of: "The mathematical or logical langauge of the nervous system is not the same as that we use when we do/talk about mathematics."
I can't decide whether the quote is vacuous or profound. From a modern perspective, we may say that the nervous system is a biological neural network, with (as von Neumann himself suggested in this book) low precision but high reliability. And then, presumably, this neural net is Turing-complete, and the actual way in which computations are performed is not important - they are still computations, and they could just as well have been run on some sort of a cellular automaton.
So my question is: are there any concrete consequences of von Neumann's observation within the realm of metamathematics? For instance, is there some theorem that becomes relevant? If the answer is 'no, this is mathematically irrelevant' then I suppose this is a philosophical question, in which case I'd be interested in implications for the philosophy of mathematics.
Level of my question: I am somewhat familiar with various notions that may be relevant, like first and second order logic, the Entscheidung problem, Gödel's theorems etc, but I was trained as a physicist, not a mathematician.
TLDR: was von Neumann onto something fundamental here about the nature of mathematics as a language , or is it merely a subtlety around the physical instantiation of systems that compute, and their architecture?