In my opinion they only mods I would do is off the shelf for a couple of reasons. First of all, the mods may have to be Coast Guard approved for use on a boat. Secondly having to be towed to shore is not good. Boats are like airplanes, reliable is the better than speed.
If you want a boat that you can modify like a car I would look into a old jet boat that has an open engine bay. Then you can get off the shelf turbo and blower set ups that are Coast Guard approved. Other than that, I am a person that says leave the boat stock.