We developed an ability to discover scientific names in millions of books in a matter of hours. We are now able to finish name-finding for Biodiversity Heritage Library (200 000 volumes) in 12 hours on a single laptop, and HathiTrust Digital Library (18 million volumes, estimated 10% of all published books) in 9 hours, using 50 servers. Scientific name-strings are often represented differently by different sources. For example Homo sapiens can be written as Homo sapiens Linnaeus, 1758, Homo sapiens sapiens, Homo sapiens Linn. etc. To understand that all these variations talk about the same species they need to be normalized. We built a tool that normalizes 300 million name-strings per hour on a single computer. Also we created a service that is able to verify scientific names in all existing variations. The service validates names against well-known curated data sources with a rate of 800,000 per hour. As a result of all these new developments we have an opportunity to create a truly global index of scientific names and mine biological information using scientific names as anchors.

Modern biology depends heavily on scientific names. They were introduced in 18th century by Carl Linnaeus. For more than a quarter of a millennium they are the basis of sharing information about biological taxa. Scientific names are heavily used by researches, physicians, students, governmental agencies, citizen scientists etc. Scientific names create a common basis for biology and give us an ability to connect millions of medical and biological facts together into a graph of knowledge.

However scientific names change quite often. According to our observations there are about 3 different scientific names per species on average. On top of that, scientific names usually are not rendered exactly the same way in different publications, as we showed above, and such lexical variants of names have to be reconciled. To collect a complete history of studies about species we have to be able to take in account of these differences between name variations. To provide an access to globally accumulated biodiversity knowledge we have to scientific names in literature published in the last 250 years.

Our group have been participating in Global Names Architecture project since 2007 and tried to find solutions for these three major questions:

• How to find scientific names in texts?
• How to “normalize” lexical and spelling inconsistencies in names belonging to the same lexical group?
• How to connect a given name to all names ever given to its taxon, and find which one of these names is currently accepted as valid?

Former ABI innovation grant from NSF from 2011 (#1062387) gave us an opportunity to develop functional prototypes of programs that were answering these questions. However, these prototypes had been too slow to be applied to whole corpus of biological and medical human knowledge.

We understood that for such programs to be useful on a global scale they must be dramatically improved in speed, quality and scalability. They must be able to use all resources of a multi-processor computer, and scale to multiple machines with ease. Current ABI Development funding allowed us to achieve speed and scalability goals.

We developed 3 major software tools:

Parser of Scientific Names: gnparser

This tool allows to dissect vast majority of scientific names into their semantic elements. For example it is able to find out that “Carex scirpoidea ssp. convoluta (Kük.) Dunlop” and “Carex scirpoidea Michaux var. convoluta Kükenthal” both have the same conservative element “Cares scirpoidea convoluta”, but vary in rank (variety vs subspecies) “Carex scirpoidea var. convoluta” and “Carex scirpoidea subsp. convoluta”. Such matching is very important for aggregation of biodiversity data from a variety of sources. The gnparser program is designed to be small, easy to install and be blazingly fast. It is currently used by several prominent biodiversity aggregators, such as Encyclopedia of Life and Catalogue of Life, as well as Global Names Index. It is able to process 300,000,000 names in an hour on one computer.

Global Names Index: gnindex

The gnindex service aggregates scientific names from a large number of biodiversity data sources and uses them to verify incoming names. A user is able to send a list of name-strings and find out if they are known names, if they contain spelling mistakes, if they are currently accepted as valid names, etc. Such service is important for many use cases, for example as a verification tool for species lists collected by citizen scientists, as a curation tool for museum collections, for mapping country species lists of government agencies to standardized data set, such as Catalogue of Life etc. This tool relies heavily on gnparser to align data from many sources. It is also indispensable for verifying (reconciling/resolving) name candidates found in texts by our algorithms. This service currently has a throughput of 800,000 names per hour.

Global Names Finder: gnfinder

The gnfinder tool uses heuristic and natural language processing algorithms to detect scientific names in texts. In machine learning terms it belongs to named entity recognition class of tools. It marks parts of texts that potentially can be scientific names, and then sends them for remote verification to gnindex. Average book in digital libraries is about 300 pages long. On a single machine we are able to reach a rate of name-detection to 45,000 books an hour.

To summarize, these new tools give us ability to create indexes of scientific names in a matter of hours for gigantic collections of literature and make a global index for all digitized literature possible.