Those can get seized up pretty bad. I once had to bang one off using a hammer, marring the shaft, then cleaning it up and grinding down the set screw from inside. I then drilled and tapped another hole and used stainless screws. Anti-seize is also your friend.