M1A's are great rifles but they carry a premium price. You can build a plain old Remington 700R .308/.300WSM bolt into a very capable rifle on the cheap. You can find Mauser 8mm or Lee Einfield .303 bolt rifles for next to nothing on Gun Broker.com.
Dropping X,XXX on a large bore rifle for fun and not having the experience/training to take advantage of it is a bit like dropping XX,XXX on an exotic and driving the speed limit with it.