I like the trunk for when I'm going on long rides, but I mainly keep mine removed. I like the way my bike looks better without it, plus it rides like a totally different bike without the trunk.
I ride within my skill set, you can't control what the idiots around you are going to do. I've had a plenty of close calls that made me think twice about riding, but I love it too much to give it up.