Yes, I know all that. The question is can this problem be overcome? Can you refuel with small access ports that won't compromise structural integrity? Fuel rods and control rods themselves are tiny, the coolant is just water. Civilian reactors don't need to worry about keeping the access ports small, but since warships do, can modern technology allow the design of smaller access ports?
Looking at this video, I don't see why the access port can't be made to be as small as basically just a little larger than a fuel rod and it seems like much of the process can be automated. If this can be done, there's also no reason why it wouldn't be used on ships that use HEU either. It would obviate the need to cut large holes in the hull or deck for refueling, even if it's not a big issue as it's only needed to be done once.
During refueling you have to worry about the radiation. If I place a single freshly irradiated fuel rod on the highway and you drive over it at 65 mph, you'd die from the radiation accumulation. So, it is not about physically fitting the fuel through, but also shielding requirements. In commercial reactor refueling, they flood the core and refueling cavity and the fuel always stays >~10 ft below water surface. You can probably get away with 5 ft of water without getting too much dose.
Control rods typically are attached to the core internals (they are not normally replaced and do not come out) but can be inside the fuel if they need be replaced.