Nope, a GM can't do anything about it. You're stuck with it on that character.
Another point with the "owned by" talismans is that they carry the "Lord" or "Lady" title if you have the title when you first equip it, but if you don't, it won't pick it up later on.