Don't know what you mean by "jam up", but if you run low speed heavily weighted, the more displacement the better. What we do on the water requires torque at low RPM. Small blocks are torqueless wonders and require gear reduction transmissions and small pitch props so they can spin to the moon to do teh same work a big block can happily do at 1000-2000 RPM less.

As for octane... An engine requires a certain octane rating fuel based on dynamic compression ratio, chamber design, and several other factors. Bottom line is if it needs 87 octane you are just wasting money putting anything higher in it. If it needs 89 or higher and you put 87 in it, even just once, you may hammer your rod bearings, crack rings, or even crack pistons. Use the required fuel for the engine. If you're unsure (old boat, non factory engine, blah blah) just put higher octane than you think it needs if in doubt. A new boat like you're talking about, just use the fuel the engine manufacturer calls for... Likely 89 octane but possibly 87.