Try latex gloves under your riding gloves. Makes a huge difference. I think it's a wind chill thing.
That said, if you do decide to get gloves you will need a controller and will find you want the jacket liner because plugging in the gloves to the liner is a hell of a lot nicer than farting around running wires down the sleeves of your riding jacket.