Any mathematical objects that may seem unconnected to reality, weren't being dreamed of a hundred thousand years ago. Modern mathematics is one of the pinnacles of social evolution and it can all ultimately be traced back to the jungle. If mathematics makes any sense at all it is somehow, however unclear at the moment, if it was built by following one logical step to the next, connected to the real world.
The discovery and early work with matrices is a good example. Very abstract initially but led to many important applications approximately a hundred years later. Many of the abstract paths we have gone down began with a desire to find better ways of proving results and solving certain problems that were themselves initially real world problems and have often succeeded in just that. Sometimes there's an element of losing ourselves in abstractions only to find ourselves with a better understanding of reality at a later stage. And Einstein, using some of Minkowski's abstract mathematics, solved a hugely significant real world problem for which previously used mathematics models were inadequate. Did Minkowski expect that when he developed that mathematics?
Historically, there was a reality before any assumptions about it existed. Simply put, humans proceed in the world by making, and testing assumptions. The progress they make is a reflection of the voracity of their understanding. But, that doesn't make reality work. Reality works, however reality does, we just discover how it works.
This is not to say that our actions don't change what exists in reality, but we don't change the how, by which I mean, for example, we can't change the laws of physics.
The original statement is about maths being built on assumptions. Well it's not just assumptions, but assumptions that turned out to be correct. Once they're demonstrated to be correct, they're not just assumptions any more. The Sydney harbour bridge is not built out of assumptions, it's built out of the real stuff that we assumed correctly about.
Even the purest of maths, no matter how seemingly removed from reality is not built on arbitrary assumptions, it still "works" somehow, we don't just make it up as we go along. However complex a structure maths has become, it is connected to the way we think, it is in fact in large part a model of how we think and, as history has progressed, its increasingly complex twists and turns reflect how our thinking has evolved. Underneath all that we strive for it to make sense, even if it may not have an obvious real world application at the time.
Thinking is real. It takes place inside real existing minds. So, my answer is a resounding no, it's not all based on assumptions it's just the pinnacle of how we have come to think.