That's always been the problem - not the people you're racing, but the pack of idiots that follow you out and find their way into the mix by zipping around like a swarm of idiotic, bad driving bees.
Street racing is, by definition, kind of stupid - and anyone who has ever done it will usually acknowledge that. However, there's a "less stupid" way of street racing, and a "more stupid" way as well.