I think most would agree that it is mostly the thumb warmer that is the most annoying when on high. The grip warmers are also easier to adjust while driving. I did what I did on my last 5 sleds. I added a switch to turn off the thumb warmer. I leave it off unless I need it.
If I wasn't brain dead, I'd do what Buddah does. He adjusts them when he starts the sled, and before driving it. With the switch though, you don't have to do that.