We have been treating Neanderthals here as a species, Homo neanderthalensis, distinct from our own species, Homo sapiens. Some researchers elect to call Neanderthals a subspecies, Homo sapiens neanderthalensis, and classify modern humans as another subspecies, Homo sapiens sapiens.
The line between subspecies and species is not clear cut, nor – given the way evolution works – should we expect it to be. Very recent work on ancient DNA recovered from fossils has shown just how complicated the subject is. The spectacular finding of the last few years is that modern humans are hybrids, getting most of their ancestry from a single founding population (we can call them Homo sapiens), but incorporating limited ancestry from close relatives. Thus human beings outside Africa have 1-4% Neanderthal DNA. So it looks as if early in the course of expansion(s) out of Africa, there was a limited amount of interbreeding with Neanderthals.* And not just with Neanderthals. Populations in Melanesia get an additional 4-6% of their DNA from a widespread East/Southeast Asian population known as Denisovans, while some African groups have ancestry from non-sapiens populations in Africa. (The fossil record for Denisovans is a lot sparser than for Neanderthals, and it’s even sparser for African non-sapiens.)
This isn’t reason enough to put Neanderthals and sapiens in a single species: plenty of species occasionally hybridize with related species. And in fact the DNA evidence implies that sapiens and Neanderthals were moving toward being reproductively isolated. Specifically, we find that a lot of Neanderthal genes related to testis development and male fertility are underrepresented (i.e. at a lot less than 1-4% frequency) in modern humans. The likely explanation is that those genes didn’t work well against a H. sapiens genetic background. In other words, if you were mixed sapiens/Neanderthal man, you probably had fertility problems, albeit not to the point of complete sterility.
There is an extensive older literature in physical anthropology on “race crossing.” Researchers were concerned with whether people with mixed racial ancestry might have reduced fitness as a result of combining incompatible genes. This literature is reviewed at book length here. The overwhelming evidence is that “race crossing” has no harmful biological consequences (in contrast to close inbreeding, which is a bad idea: wait until we get to the Habsburgs on November 9). The new data from Neanderthal DNA puts this in perspective. Homo sapiens and Homo neanderthalensis evolved separately for the better part of a million years, and were some way on the path to reproductive isolation. By contrast, different populations (“races”) within Homo sapiens have only been evolving separately for 100 thousand years or less. This has been enough time to evolve major differences in traits like skin color and hair form, but not to create appreciable biological barriers to interbreeding.
And here’s a link covering some recent research suggesting that across a wide range of organisms it takes a surprisingly clock-like average of two million years separation to split one species into two.
* Hence, my Neanderthal name is Carg, and my website is cargshome.