Why don't we subtract out / ignore the leap year every 128 years? When we add the extra day to the calendar every 4 years it's to "correct" the calendar under the assumption that the year's length is 365.25 days. Every year, you're .25 days off, so after 4 years, you're 1 day too far ahead, so add in an extra day to resync.
But in reality it's more like 365 days, 5 hours, 48 minutes, 45 seconds. In other words it's not 365.25 days exactly, we're over-stating it by 675 seconds.
Every 4 years when we add that day back we're actually adding 2700 seconds too much!
After doing this 32 times though (after 128 years), we'd have added 2700 * 32 = 86400 seconds too much. But this is exactly the number of seconds in a day, (24 * 60 * 60).
So why aren't we subtracting a day back out every 128 years? Instead we're subtracting the day out after 400 years instead.
 A: You asked:

So why aren't we subtracting a day back out every 128 years? Instead we're subtracting the day out after 400 years instead.

The rule in the Gregorian calendar subtracts out the leap day 3 times in 400 years. (The leap days in years 1700, 1800, and 1900 were skipped; 2000 was not skipped, and 2100 will be skipped again.)
So we subtract out a day, on average, every $\frac{400}3$ years, and guess what, $\frac{400}3≈133.3$, which is quite close to the 128 you suggested.
The remaining discrepancy can be explained:


*

*At the time the Gregorian calendar was devised (1582 or thereabouts) the length of the year wasn't known with such accuracy.

*The length of the year varies slightly, so there's no point in fine-tuning the correction past a certain point.

*It's more important to have a simple rule than a perfectly accurate one. Whether the calendar loses a day every 3000 years or every 5000 years is a minor issue compared with the trouble of retraining everyone.


If you do want to fine-tune the calendar, I recommend 8 leap days every 33 years.
A: The only reason I can think of is a practical one: it is simpler (in terms of the numbers involved) to treat as special those years which are divisible by $4$, $100$ and $400$ (since they are easy to determine), than to do it for those divisible by $128$. It also keeps the maximum error below $6h$ (*), quite below $31d$, which is approximately what you would get before the $128$th year correction... an error comparable to the one provoked by the Julian calender, unacceptable in modern terms. In fact, to correct that difference, we should add a whole month every $128y$.
But if you're asking why don't we have leap years every four years, as we do AND correct the excess every $128y$, once more the answer is about keeping it simple.
(*) Keep in mind that every year divisible by $100$ is not a leap year, unless it is divisible by $400$.
It is good to know, somehow, that being close to the year 2000, we have put the things in their place quite recently... so to say. (?)
ADDITIONAL NOTE: It is also interesting to note that, with the Solar System being a complex and —after all chaotic— dynamic system, the actual length of the Earth's orbit around the Sun and its relation to Earth's rotation varies from year to year in a not so predictable way. That's why after determining this actual length for the last years astronomers decide if there is need for some adjustment and, in consequence, from time to time a leap second is added after the last second of the 30th June or the 31st December.
A: One issue - this is a complex issue going back to when the modern calendar was created - is convenience, at least for the layperson.
It's exceptionally easy to remember and apply the current system, even if said system is complicated. 

"Add an extra day on years which are multiples of $4$, unless they're multiples of $100$, unless they're multiples of $400$."

Divisibility by each number is easy! Numbers divisible by $4$ can be shown to have their last two digits divisible by $4$. And then the only complications occur every century, and divisibility by $100$ and $400$ are equally trivial: end in $2$ zeroes to be divisible by $100$, have third-and-fourth to last digits divisible by $4$ as well for divisibility by $400$.
Meanwhile, on the other hand, for a layperson, divisibility by $128$ ... not so easy. The only method that would have any simplicity that comes to mind is just dividing by $2$ seven times ($2^7 = 128$) and if at any point you get a remainder, then no extra year. 
The issue being it's not immediately apparent to someone, when if you're somehow not sure it just takes a few seconds under the current system to check: for the "better" system, while more accurate, most would have to break out a calculator. (And I imagine that $128$ is not exact enough and would cause overages/shortages every few millennia or so in its own right, which would likely make it even harder.)
A: A long term day-of-the week advantage exist to the 400 year rule.
The Gregorian calender design also had day-of-the-week for a given year as a priority (When did Easter, a Sunday, occur for a given year?) as well as preference for base-ten math.
The standard rule of 97 leap days every n=400 years has a subtle advantage over OP's rule of 31 leap days every n=128 or this answers suggested 8 leap days every n=33.  The day-of-the-week  pattern repeats n years is the first, but not in the other patterns.
In other words, having a leap days every b years should have 365*b + a or b+a as a multiple of 7 to ease day-of-the-week calculations.
