Seem to remember your skill level had to be close to the skills of the golem to get gains. Not sure of number of hand but lets say that if golem has 85 you would probably need to be in high 50's to be able to gain.
Also believe you should be in a guild when working with a golem