We have been treating Neanderthals here as a species, Homo neanderthalensis, distinct from our own species, Homo sapiens. Some researchers elect to call Neanderthals a subspecies, Homo sapiens neanderthalensis, and classify modern humans as another subspecies, Homo sapiens sapiens.
The line between subspecies and species is not clear cut, nor – given the way evolution works – should we expect it to be. Recent work on ancient DNA recovered from fossils has shown just how complicated the subject is. The spectacular finding of the last few years is that modern humans are hybrids, getting most of their ancestry from a single founding population (we can call them Homo sapiens), but incorporating limited ancestry from close relatives. Thus human beings outside Africa have 1-4% Neanderthal DNA. So it looks as if early in the course of expansion(s) out of Africa, there was a limited amount of interbreeding with Neanderthals.* And not just with Neanderthals. Populations in Melanesia get an additional 4-6% of their DNA from a widespread East/Southeast Asian population known as Denisovans, while some African groups may have ancestry from non-sapiens populations in Africa. (The fossil record for Denisovans is a lot sparser than for Neanderthals, and it’s even sparser for African non-sapiens.)
This isn’t reason enough to put Neanderthals and sapiens in a single species: plenty of species occasionally hybridize with related species. And in fact the DNA evidence implies that sapiens and Neanderthals were moving toward being reproductively isolated. Specifically, we find that a lot of Neanderthal genes related to testis development and male fertility are underrepresented (i.e. at a lot less than 1-4% frequency) in modern humans. The likely explanation is that those genes didn’t work well against a H. sapiens genetic background. In other words, if you were mixed sapiens/Neanderthal man, you probably had fertility problems, albeit not to the point of complete sterility. On the other hand, other Neanderthal genes – especially genes related to immune function – were useful to modern humans moving into Neanderthal territory and are found at high frequency in Europeans today.
There is an extensive older literature in physical anthropology on “race crossing.” Researchers were concerned with whether people with mixed racial ancestry might have reduced fitness as a result of combining incompatible genes. This literature is reviewed at book length here. The overwhelming evidence is that “race crossing” has no harmful biological consequences (in contrast to close inbreeding, which is a bad idea: check out this post on the Habsburgs.)
Some new data put this in perspective. My colleagues Alan Rogers, Chad Huff and Ryan Bohlender have just shown that the population ancestral to Neanderthals and Denisovans separated from the ancestors of Homo sapiens somewhat more than 750,000 years ago, passing through a narrow population bottleneck, probably in an early Out Of Africa episode (before the later modern human Out Of Africa episode). This population then split somewhat less than 750,000 years ago into Neanderthal and Denisovan branches. So Homo sapiens and Homo neanderthalensis evolved separately for the better part of a million years, and were some way on the path to reproductive isolation. By contrast, different populations (“races”) within Homo sapiens have been evolving separately for 100,000 years or less outside Africa, and perhaps 250,000 years within Africa. This has been enough time to evolve major differences in traits like skin color and hair form, but apparently not to create appreciable biological barriers to interbreeding.
And here’s a link covering some recent research suggesting that across a wide range of organisms it takes a surprisingly clock-like average of two million years separation to split one species into two.
* Hence, my Neanderthal name is Carg, and my website is cargshome.