If the vendor were to run out of gold all it's items simply go to the house's moving crate to be reclaimed. Also BOD books cost almost nothing to keep on a vendor even when fully priced with expensive BODs. With nothing but BOD books only, a vendor only charges a fee of 60 gp per day so 5k will get you almost 3 months of vendor fees.
As for storing BODs on a vendor, it's simple. Put a BOD book with unpriced BODs onto a vendor and press 'esc' so as to not put a price on the book. Be sure you do this otherwise the entire book can be purchased.
I'm usually extra double careful and press 'esc' a couple times when placing the books back onto my vendor. Then I mouse over the book to check it's properties and make sure it's "not for sale." I have however, twice that I can remember, opened up a vendor's backpack to discover a book missing. Luckily it was only low level reward BODs both times.
Any BODs that aren't priced cannot be purchased from the book and as long as you pressed 'esc' when you put the book into your vendor's pack the book cannot be purchased either.
I have 28+ BOD vendors and 4 vendors used just for storage in between restocking. Before BOD books and storing on vendors, the BODs on those 4 vendors alone used to take up nearly 2000 lockdowns.