Nearly 1200 fishing quest results tracked;
https://docs.google.com/spreadsheet/ccc?key=0AiNhz6t0aNtRdDZKYURSUWpEd0JFekJfNHhDU01fUlE#gid=0
So to answer the question... it's random, but the odds are very, very low for a 120 scroll, and even lower for some of the rarest bait. My best guess is the code is doing a double roll; first for type, then for rarity. If someone knows how to code the document so as to pull the exact percentage for an individual reward type, please add it to the script... also add your own results if you wish.
As for what cancelling a fish quest does; reputation works like this.
* For every quest you take, and succeed at, the fish types in that quest raises by 5, so it goes 10 fish, 15 fish, 20 fish where it caps
* For every quest you cancel, the fish in the cancelled quest drop by 5, so 20 down to 15 down to 10 fish, where it caps again.
* The quest rolls number of lines independently of skill, or reputation
The system continues like this, we know for sure
* The quest rolls type of fish based of having 106+ skill (requiring a 110 scroll then) for dungeon fish in the roll. Otherwise this is also random.
* Number of fish is then taken from your "reputation"
* Thus dropping quests limits the maximum size of the quest, IF you roll for the fish you left behind. Otherwise it has no effect. And you can still get a 1 liner for Shore Fish no matter how much fishing you do.
This bit is my intuition from the sample size of results I have:
* Rewards are point based from the size of the quest, but the reward table is a range of percentage chances not a guarantee. Items are always 1x, Bait is always identical to quest point total, until it hits the cap of 100 points, and Charbydis Bait however is always only 5 units of bait. That is known. What I'm not sure of is how it's deciding the spread of quality for what you actually get.
You can abandon every quest bar for 6 Dungeon fish then, and still theoretically roll a 6 liner which contains only those six fish, and thus have a maximum point quest all the same... and it can roll "Type: Bait" "Intensity: High" for reward and you'd just get 100 Legendary Fish bait again.
You will do this alot.
Charbydis bait is I think the Elite (my words) level reward, and it's turned up about 4 times in 1200 attempts? I've not looked at the document for a few months I'm afraid, when 900 results were largely me with a few inputs from a friend (Now someone else is adding them, hurrah!) but it was the same frequency roughly as the 120 scroll when I did. Which is why I suspect that it's going type, intensity and a spread of percentage chances for the best reward types.
So the conclusion? Don't worry about "reputation" at all really. It helps a little, but you'll muff the rolls so many times anyway, even when handing in a monster of a quest (and you can see from the records I did that teeth gnashingly many times) that any fish you drop will likely have been pushed back to 20 again long, long ago...