The only "housing" issue is with players who have to have the house they are in now, in the location it is in now, for most this only consists of the few luna house holders and people with castles (maybe keeps). Easy solution for the luna situation is, remove housing rights from luna there no more unfairness. Castles and keeps....I do not have a current solution for. As for all the other housing, Put every single house into an "instance" like a giant bank box that you can never add items to, so all the items in your house can stay there and once you place a new house you can always move all the items out of there, so there you have it housing is solved. As for the people with "servers moving" if you play on pacific or napa and sonoma you will notice you ping is always the same...because the servers are so close together that the change in your ping is below the scalable amount, so if you ping 10 to pacific, you might ping 10.1 to napa. This low change in ping is not noticeable, you can in fact live with a ping of 1-10 higher than your current ping and most likely never notice the difference.
Any other arguments other than the castle and keep people? This is not likely to ever happen, Altho it would help considerably with the "community" issue everyone has, and lets no one forget the number of shards we have now has MULTIPLIED over the years (something like 5 times the number of shards compared to 1 year after launch?)
I would be all for a strategic merger of the servers, but I do not think they are capable of pulling it off in a pleasing manner