It's the same distance as you'd need to be from the coast.
However, Server lines are a bit more vague, as the game actually won't let you come within the length of your boat's usable space to the actual line (at least, if perpendicular to the line - it may go be width if parallel to the line - not tested).
On a HS ship of any type, you'll end up jumping over an entire degree as you cross, and that can actually put some SOS locations in the no-man's land between the margins (though you'll still be close enough on either side to fish it up). If you've not become accustomed to it, this can drive you bonkers, when using one of the map programs (you get within 4 or 5 spaces to a SOS, then your ship jumps at least as far past it). You can get closer on a classic ship, and even closer on a rowboat.
Strangely, I've noticed that if my ship is straddling the deep/shallow margin, if I cast in one area, then walk into the other area before the fish comes up, the fish type will be based on the location I am standing at when the fish comes up. Example: I make two casts near a server line, get shallow water fish. I cast from the same spot to the same target water, but move to the other end of the boat before the fish comes up - and I get a Red Snook (or other deepwater fish). So, while nets are based on the targeted water square, casting appears to be based on where you're standing when the fish is caught.