The limit is 2 billion gp, as the amount of gold on a vendor is stored in a 32-bit signed integer, which has a maximum value of 2,147,483,647. (This is also why gold piles max out at 60,000gp: each gold pile stores its amount in a 16-bit unsigned integer, which maxes out at 65,535; 60,000 is much more pleasing to the eyes.)
It would seem the devs of the distant past were smart enough to place a programmatic limit at the largest round number before the integer limit, as otherwise, the number would either overflow, or wrap around to -2,147,483,647.
Unfortunately, they were not smart enough to make it so that it immediately prevented you from breaching this limit. Example: I have 2,000,000,000 gp on my vendor. I sell an item off it for 111,111gp. I now have 2,000,111,111gp on my vendor. Server down hits. I come back the next morning. I now have 2,000,000,000gp on my vendor. The server ate the other 111,111gp to keep me under the limit. I'm not sure if this bug was ever fixed.