There is an very interesting paper in Nature (doi:10.1038/nature12961) about the Neanderthal genom part in modern men. It shows an a chromosome by chromosome demonstration how much of the Neanderthal ancestry is found in the autosomes and in the sex chromosome: about 1 % in autosome and considerable less in sex chromomes. In the Y-chromosome it is barely found.

When the two populations first encounterd some 50000 years ago (as a paper form PLOS suggested: DOI: 10.1371/journal.pgen.1002947) we can envisage how this was done:  there was war and the Neanderthal female were raped after the man were killed. Whether this was a singular event or repeatingly done in the clash of general war betweem the two species is not documented, but when the progeny e.g. was selected for the “African” phenotyp then the Neatherthal would be not maintained in the Y chromosome.

Very suggestive! And much has not changed till modern times