When I am talking on the phone and driving I will put the phone down if I need to. Some asshole almost cut me off I will drop the phone mid sentence, move my car so I don't get hit, flick off asshole, set cruise control and pick the phone up.
It isn't that hard if you make driving your top priority over everything else.