The speed of the ride leader, or other riders in NO WAY should affect your speed.
Your speed should be based on your skill, confidence, and how you prefer to be riding that day. End of story.
If I'm content following the rider in front of me, I'll stay there. If not, I may pass in a straight, or the next time we change roads. If I want to chill, I'll either drop way back, or let someone know that I'm going to split off and do my own thing. In any case...I am riding MY ride.