I got two relic fragments from 6 hours of farming turkeys (a few minutes each) at 400 Luck, two relic fragments from an hour at Swoop with 2000+ Luck. This is from unraveling bags of pure loot, though.
I have only just recently begun properly sorting through my bags of loot and enhancing items for higher return. A few pieces that are almost a relic fragment can be improved with some special material, if you succeed. You lose a few, but you're only burning enchanted essence and some material.
I find this way a slower pace than just dumping mass bags of loot and unraveling everything at once. However, it yields more relics for less hunting, which is my preferred playstyle.
One thing that has made the entire process so much easier is the Item Properties mod for Enhanced Client. You can mouseover and see all the properties and their intensity colour-coded, plus it lists item resists next to your current resists which is helpful for building suits. It also tells you the overall intensity of the item, and its unraveling worth based on your race and soulforge. If something can be enhanced to upgrade it to a relic fragment, it'll say what minimum material is needed.
As for using semi-useful loot and imbuing from there, I find that works best with jewels. For other stuff, I prefer exceptional items. I generally make my own jewels from scratch but I stockpile useful pieces of loot for one-offs.