This happened aboard a Tokuno ship with the classic client. If you stand at the very front of the ship and double-click wheel to take control, and then turn the ship east or south, you can't see the wheel on your screen anymore, even with object handles up.
If your ship were then boxed in by other boats and couldn't turn - and you'd used another item so a "lastobject" macro wouldn't re-use the ship's wheel - there seems to be no way to relinquish control of the ship. Even logging out and back in wouldn't remove me from mouse piloting mode.
EDIT
To BTeng: Thank you, that worked. I just double-clicked myself and was able to release control. I must have overlooked that part when reading about the boat controls - looks like the only problem here was my own confusion.