Yes your are right. I would definitely check out all the easier stuff first. Loose connections, faulty relays, bad grounds, oxidized fuses, etc. Then change your starter, :). I have been in the same position many times. You kind of know what it is but you are hoping it's not so. lol