Some tests seem to indicate it's better to use a talisman with only an exceptional bonus - but that could just be superstition combined with weird runs on the RNG. The normal bonus from the talisman isn't SUPPOSED to affect skill gain, but I personally wouldn't risk it.