Honestly, gems are the least problem. The gems you need for imbuing cost around 50 gold a piece. If you buy a bulk of 500, that's 25,000 gold coins. It is much faster making that gold by hunting creatures, than collecting the gems by yourself by hunting creatures or searching for treasure.
It is much more difficult to collect the large number of Magical Residue you will need to train Imbuing. I'd recommend to buy several unraveling bags, collect every piece of monster loot you can find, bring the full bags back to your house and unravel them when you have a couple of those.
In 1 hour a skilled fighter can earn about 50,000 gp AND collect 3 or 4 bags full of monster loot for unraveling. (Which yields 1000 gems and about 400 Residue.)