It isn't "just". Speed comes with:Speed just means you don’t need to spend a lot of time in places you shouldn’t be hanging around in.
1, less effective plane forms for radar stealth, much higher stability requirements, problems with RAMs and more required air flow to engines. You're outright less stealthy, and given how finicky VLO stealth is - we're speaking about losing orders of magnitude in stealth(i.e. detection range will increase several times). Your "places you shouldn't be in" just get larger.
2, you're turning much less efficient i.e. you aren't nearly as free in choice of optimal flight paths around defenses.
3, you're much brighter, and worse still, you are suddenly not just somewhat visible in LWIR, but also appear in MWIR. Which means every f-35, J-16 and likely J-20A/35 around.
4, and how much speed are we talking about? Other than Hustler, pen speed for majority supersonic heavy bombers was(is) at most M=1.4. It's not that terribly faster than M=0.95. is this x1.5 really worth 2-3-4 times increase in danger spaces?
And for all this, you pay a lot.