Just to be a little more clear on that, the shaft is actually FULL, so there is no room for the bearing to move, even without a clip. In place of the clip, there is a white vinyl sleeve.
And yes, you can use a wooden mallet (or similar) to knock the wheel off the shaft. Don't hit too hard though, you dont want to damage the wheel.