The brokers only transfer funds between bank boxes; carried checks and gold will not be used. It is currently possible to price a pet higher than someone should theoretically be able to pay. The brokers will happy collect their fees for your impossibly priced merchandise...
If the broker's balance goes negative, the owner is not allowed to do anything with the broker until the balance is brought positive again. You are still allowed to bid on stalls, though, so if you win a new stall while your broker has a negative balance then the broker will still transfer to the new stall, keeping its current state.
A broker with a negative balance will still attempt to do business with customers. The proceeds from any sales will be first applied to the back fees.
If a broker with a negative balance goes to storage, you can't retrieve items from storage until you pay the back fees. If you get yourself into too deep a hole and can't or won't pay the fees, then after 7 days in the warehouse the broker inventory will be deleted and the back fees written off.
These brokers assess their fees on a continuous basis instead of assessing them periodically. There is no way to avoid the fees by attempting to re-price inventory immediately before an assessment, so there is no need for a penalty. When you re-price inventory, the old fee is assessed up until the moment of the re-price, then fees are immediately recalculated. The same is true for sales; the fees are assessed up to the moment of the sale, then fees are reduced immediately at the time of a sale to reflect the decrease in inventory value.