How Do We Define A Species?

When it comes to species, says biological anthropologist Rebecca Ackermann, “forget everything you learned in high school.”

The classic textbook definition, known as the biological species concept, is a group of organisms that only produce fertile offspring with one another. By this rule, domesticated dogs are a single species — whether dachshund or Great Dane — but a donkey and a horse are not.

Ackermann, a professor at South Africa’s University of Cape Town, favors a different definition that’s not dependent on successful sex: a group of organisms sharing a mix of anatomical, behavioral and genetic traits that distinguishes them from other groups. But, she adds, “Many, many evolutionary biologists I know often avoid the word species entirely.”

That’s because the evolutionary tree is tangled, and many organisms on diverging branches can still interbreed. “Canids have, pigs have, mice have. You name it, and it has,” says University of Georgia evolutionary biologist Michael Arnold. “There’s a hybrid under every bush.”

That includes our ancestors. Genetic evidence has shown that ancient Homo sapiens interbred with Neanderthals and their eastern cousins, Denisovans, several times from 100,000 to 40,000 years ago. Neanderthals and Denisovans mated with each other, and Denisovans hooked up with a distant lineage, not yet known from fossils, that may have been closer to the earlier Homo erectus.

In recent ancient DNA studies, almost “every time a new individual is sequenced from the human fossil record,” says Ackermann, “there’s some new piece of evidence for gene flow.”

Interbreeding may have been a common theme in human evolution, but it’s difficult to understand today, when ours is the only Homo species left. To better grasp our past, anthropologists like Ackermann have begun researching other animals that mate across classic biological species lines.

There is one major problem with the biological species concept: It can’t be applied to organisms that reproduce asexually, which includes most microbes. The sex requirement also poses a challenge for extinct animals because paleontologists can’t tell just by looking at two similar fossils if the creatures could mate. Excluding extinct and asexual organisms means the biological species concept does not work for the vast majority of life that has existed.

Nevertheless, biologists long championed the definition, in part because it fit the prevailing image of evolution as a branching tree, in which a single ancestral species diverges into distinct lineages of descendant species.

“That was a classical idea,” says Arnold. Instead, he says, we need to recognize that as descendants branch off, “genes are still being exchanged, even though we call them different species.”

Interbreeding has now been detected in 10 percent of animal species and 25 percent of primates, including ongoing crosses between such distant relatives as gelada and baboon monkeys, whose last common ancestor lived about 4 million years ago.

Recognizing the ubiquity of interbreeding among primates, Arnold authored a book a decade ago that argued hybridization likely occurred in human evolution. It received some scathing reviews from anthropologists, who thought Homo sapiens took no part in interspecies affairs — until several months later, when sequencing the Neanderthal genome revealed they did.

“I tried not to feel too smug because I’ve been so wrong about so many things during my scientific career,” Arnold recalls.

Although DNA confirms that ancient hominins interbred, it doesn’t tell the whole story, including how hybrid offspring fared in different environments or societies.

“We’ll never be able to actually watch interactions between archaic hominins and our own lineage in real time, but that’s something we can do with nonhuman primates,” says Jenny Tung, a biological anthropologist at Duke University.

Tung studies baboons in Kenya’s Amboseli basin. The population has been observed for decades. Although it’s mostly yellow baboons, in the 1980s researchers noticed occasional anubis baboons, a separate species found to the north.

The lineages split roughly 1.4 million years ago — more than twice the time thought to separate Neanderthals and modern humans. But despite physical and genetic differences between the species, hybrids documented among the baboons are fertile and thriving.

“That started raising a lot of questions about how can these two parent species stay separate from one another if they appear to be admixing pretty freely,” Tung says.

To understand this process, Tung and colleagues have been conducting ancestry tests — “We watch, you know, Abby the baboon poop, and then we pick up her fecal sample and extract DNA from it” — and comparing those results with behavioral observations.

In the team’s recent study, the 22 Amboseli baboons tested had 12 to 72 percent anubis ancestry, including individuals that looked like pure yellows. The pervasive anubis genetic signatures suggest the species have been mating for hundreds, even thousands of generations.

Further analysis, including baboons from farther afield, however, showed hybrids exist only in a narrow, transitional zone between the core territories of anubis and yellow baboons. It’s possible, Tung thinks, that hybrid traits in this particular habitat or social group may not be beneficial elsewhere. For example, the darker anubis fur that some hybrids have may overheat baboons in pure-yellow territory, which tends to be hotter, drier and more savannah-like compared with mountainous forests of pure-anubis land. If so, it suggests that hybridization between hominins may also have been useful in some environments and societies, but not others.

Ackermann also researches baboons, but hers are dead. An expert in skeletal anatomy, she got interested in hybridization in the early 2000s while studying bones from baboons with known pedigrees. She’d notice that some individuals looked a little unusual, only to check their records and find they were hybrids.

“Very quickly I decided I was going to try to characterize quantitatively what I was seeing,” Ackermann recalls.

She hoped to find skeletal features that result from interbreeding in general, which could be used to spot hybrids among human fossils. It was a provocative goal, years before genetic methods would confirm hominin hybridization even took place. But her approach has an advantage over even the most cutting-edge genetic analysis: While ancient DNA can reliably identify hybrids, it’s rarely preserved in bones from warm regions older than 10,000 years. Because she focuses on visible skeletal traits, Ackermann can potentially identify hybrid fossils spanning the more than 6 million-year history of hominins.

Measuring skulls from 169 yellow, anubis and hybrid baboons that had lived in captivity, she found that hybrids are not simply intermediates of the parent species. They do, however, frequently have abnormalities such as extra or misaligned teeth. It made sense to Ackermann: Developing hybrids would receive commands from both parents’ genomes, which might not be synced in terms of when certain teeth or bones should form. These little glitches wouldn’t affect the individual’s evolutionary fitness, but they could tip researchers off to a hybrid.

Ackermann found the same developmental quirks across diverse mammal groups such as wildebeests and gorillas. Then she applied her method to human ancestors.

Anthropologists have long debated whether particular hominin fossils could represent hybrids, but usually based their arguments on the unfounded assumption that hybrids resemble intermediates, or averages, of their parents. In a 2010 study, Ackermann used developmental anomalies to suggest hybrid candidates, such as 130,000-year-old Neanderthals from Croatia with abnormal premolars and 100,000-year-old modern humans from Israel with traits such as misaligned teeth and an asymmetric face.

She also included a 35,000-year-old anatomically modern human skull from Romania. Five years after Ackermann’s study, ancient DNA from another Homo sapiens at the same site revealed that individual had a Neanderthal ancestor just four to six generations back.

Ackermann and her colleagues want to know how common these development quirks are among hybrids, and whether they persist in subsequent generations. Her colleagues at the University of Calgary bred several generations of mice from three subspecies and two species for the project. University of Cape Town’s Kerryn Warren, a recent doctoral student on Ackermann’s team, analyzed the bones.

The project is ongoing, but Warren’s doctoral thesis, titled “Of Mice and Hominins,” offers a preview: Hybrid communities are surprisingly diverse. Offspring can resemble one parent or a blend — or take on forms unlikeeither parent. Admixture allows species to swap and mix their genes, and the novel traits resulting in these “hopeful monsters,” as Warren and other researchers call hybrids, could be useful evolutionary shortcuts, especially for our ancestors.

As Homo sapiens expanded globally, they confronted new environments, foods and diseases. The migrants could have evolved adaptations to those pressures, but it would have taken many generations.

Some groups, however, were able to speed up that process: By mating with Neanderthals and Denisovans they encountered in their new world, the lineages of the newly arrived Homo sapiens could have acquired locally adaptive genes related to skin and hair color, metabolism and immunity over mere decades. “That jumps so far ahead of what you could achieve through natural selection,” says Ackermann. “Hybridization had a serious effect, a big impact on people.”

[This article originally appeared in print as "Hopeful Monsters"]