I would challenge anyone to define "passive scanning" in a way that doesn't also imply a search without consent or probable cause. Since carrying with a permit is a lawful act(we shouldn't need a permit to exercise a right, but that's another conversation), and the police exist to enforce the law, if I am not breaking the law, then you have no business asking me anything, and doubly no business "passively scanning" me as I go about my day.
I respect that you choose not to exercise your right, but my rebuttal to your hypothetical anecdote is this. If you are "forced" to use your firearm to defend yourself as described, you are in danger of losing your life, or you are improperly using your firearm. It is also entirely possible another person would become aware of said life-endangering situation, and without prior knowledge mistakenly confuse you as the aggressor/attacker and injure/kill you. What I propose is that an order of operations need be applied to the threat(s). Now if the person/persons originally attacking you are in fact threatening your very existence, it would seem folly to prioritize the possibility of being mistakenly shot or killed over the much more palpable threat of the person right in front of you, attacking/robbing you. If you are more worried about a passerby becoming aggressive towards you, then the person directly acosting you, then you were not in the type of situation that requires deadly force in the first place.