If you're dealing with a very small projectile, the trick is, it's a grenade, not a rifle cartridge. There's fundamentally different physics involved, of which I have just become cognizant. The funny thing is, while the energies of a grenade scale up linearly with volume, the velocity of fragmentating projectiles do not. They remain constant, depending on the ratio of explosive to shrapnel.
The difference between a gun and a bomb is that with a gun, all of the explosive force of the propellant is concentrated by the metal tube to force a projectile in one direction. This allows scaling to enable the projectile to achieve even higher speeds. With a bomb, on the other hand, since your explosion is uncontained, the only thing that affects the physics of the weapon are the chemical properties of the explosive and the ratio of explosive to shrapnel. You cannot exceed the maximum speed created by the explosive's pressure.
What you get with a larger grenade, as opposed to a smaller grenade, then, is a greater amount of fragmentation and shrapnel being pushed out at the target. This results in an increase in damage, but the scaling factor is linear: a volume of 20mm grenades equal to 40mm grenades, with the same explosive-fragmentation ratio, deals equal, if not more damage. Put another way, a group of grenadiers loaded with 20mm grenades can put out a volume of fire equal to a 40mm grenade.
Think about it like atomic weapons. The largest nuclear bomb ever launched was the Russian Tsar Bomba, but the 100 MT device was never mass-produced. It was a demonstration of technological prowess, only. In fact, even with ICBMs, most use MIRVs, or submunitions, which employ a greater number of smaller warheads, which, despite being lower in total yield, are greater in destructive potential as the total area under effect increases.
The problem with downscaling your grenades is not a direct problem of power, but an indirect one. As your grenades become smaller, the quantity of electronics and casing relative to the explosive increases, reducing your penetrating potential. But I am trying to get better stats on this matter.
===
(Edit: second paragraph is added)