Wouldn't that depend on the population dispersion and how easy it is for them to mingle? For example there are some differences between say a bengal tiger and a siberian tiger. They are relatively isolated by each other under pretty different conditions that probably applied different selection pressures across the whole of the populations each, so overtime they can become isolated but different.
Are there certain selection stress that can alter the phenotype of a species without turning it into a different one? And where are those located on the DNA as opposed to genes that more strictly define intra-species compatibility?
Some maybe not extinction necessary but just have subsets of populations that later on become isolated from each other under different conditions, or an event that causes a genetic bottleneck. It's just that humans have become so well adapted and interconnected due to technological advances.
Your example is simply a unique case of eliminating the dominating species. You can kill them off, or take them away. The goal is to remove the evolutionary pressure. This is why small isolated islands, like Madagascar, have strange animals.
Definition of species is all by human. We define what should be considered as monkey, what should be considered as dog and what should be human. These differences are all artificial. There is no biological differences. Evolution does not discriminate. To nature, the only thing that matters is survive. It could be crocodiles, which have not changed in hundreds of millions of years. It also could be human, which has changed so much from our rodent-like ancestors living with the dinosaurs. We certainly look nothing like the early mammals that lived with the dinosaurs. Yet, we evolved from them. There is absolutely no pressure for or against any direction of evolution. It's all chances.
Intra-species compatibility is a highly complex matter. There is no one thing that determines it. It could be the total # of chromosomes. It could be a unique gene on the certain single chromosome. For instance, horses have 64 chromosomes while donkeys have 62. A male donkey and a female horse give a mule, which has 63 chromosome. Because of the old # of chromosome, mules cannot have offsprings.
Again, your example is just a special case of evolution. In fact, your case is the exact one that Darwin found on Galápagos Islands and allowed him to come up with the theory of natural selection.