It's between the M3 or the S4 for me of course. The M3 is sportier right out the box, but the S4 has 4 doors and quattro. The S4 sounds good, the M3's sound I dislike. Both of them have their issues.
Based on your requirements I think the S4 fits your needs better because it can fit 4 comfortably (better than the M). If it came down to it, I think I'd probably take the S4 too. It's a tough one though.
My brother has an S60R. It's a nice car, but it's no M3 or S4.