As powerful, essential and ubiquitous as Google and its search engine peers are across the world right now, needs often arise in many fields and marketplaces for platforms that can perform much deeper and wider digital excavating. So it is that two new highly specialized search platforms have just come online specifically engineered, in these cases, for scientists and historians. Each is structurally and functionally quite different from the other but nonetheless is aimed at very specific professional user bases with advanced researching needs.
These new systems provide uniquely enhanced levels of context, understanding and visualization with their results. We recently looked at a very similar development in the legal professions in an August 18, 2015 Subway Fold post entitled New Startup’s Legal Research App is Driven by Watson’s AI Technology.
Let’s have a look at both of these latest innovations and their implications. To introduce them, I will summarize and annotate two articles about their introductions, and then I will pose some additional questions of my own.
Semantic Scholar Searches for New Knowledge in Scientific Papers
First, the Allen Institute for Artificial Intelligence (A2I) has just launched its new system called Semantic Scholar, freely accessible on the web. This event was covered on NewScientist.com in a fascinating article entitled AI Tool Scours All the Science on the Web to Find New Knowledge on November 2, 2015 by Mark Harris.
Semantic Scholar is supported by artificial intelligence (AI)¹ technology. It is automated to “read, digest and categorise findings” from approximately two million scientific papers published annually. Its main objective is to assist researchers with generating new ideas and “to identify previously overlooked connections and information”. Because of the of the overwhelming volume of the scientific papers published each year, which no individual scientist could possibly ever read, it offers an original architecture and high-speed manner to mine all of this content.
Oren Etzioni, the director of A2I, termed Semantic Scholar a “scientist’s apprentice”, to assist them in evaluating developments in their fields. For example, a medical researcher could query it about drug interactions in a certain patient cohort having diabetes. Users can also pose their inquiries in natural language format.
Semantic Scholar operates by executing the following functions:
- crawling the web in search of “publicly available scientific papers”
- scanning them into its database
- identifying citations and references that, in turn, are assessed to determine those that are the most “influential or controversial”
- extracting “key phrases” appearing similar papers, and
- indexing “the datasets and methods” used
A2I is not alone in their objectives. Other similar initiatives include:
- In 2014, IBM’s Watson technology² was used to mine 100,000 papers to meaningfully be able to “predict the interactions of a tumor-suppressing protein“.
- The Defense Advanced Research Projects Agency (DARPA) is currently developing a system, due by 2017, codenamed Big Mechanism to scan scientific papers on specific malignancies in order “to identify certain treatments”. ³
Semantic Scholar will gradually be applied to other fields such as “biology, physics and the remaining hard sciences”.
BigDIVA Searches and Visualized 1,500 Year of History
The second innovative search platform is called Big Data Infrastructure Visualization Application (BigDIVA). The details about its development, operation and goals were covered in a most interesting report posted online on NC State News on October 12, 2015 entitled Online Tool Aims to Help Researchers Sift Through 15 Centuries of Data by Matt Shipman.
This is joint project by the digital humanities scholars at NC State University and Texas A&M University. Its objective is to assist researchers in, among other fields, literature, religion, art and world history. This is done by increasing the speed and accuracy of searching through “hundreds of thousands of archives and articles” covering 450 A.D. to the present. BigDIVA was formally rolled out at NC State on October 16, 2015.
BigDIVA presents users with an entirely new visual interface, enabling them to search and review “historical documents, images of art and artifacts, and any scholarship associated” with them. Search results, organized by categories of digital resources, are displayed in infographic format4. The linked NC State News article includes a photo of this dynamic looking interface.
This system is still undergoing beta testing and further refinement by its development team. Expansion of its resources on additional historical periods is expected to be an ongoing process. Current plans are to make this system available on a subscription basis to libraries and universities.
- Might the IBM Watson, Semantic Scholar, DARPA and BigDIVA development teams benefit from sharing design and technical resources? Would scientists, doctors, scholars and others benefit from multi-disciplinary teams working together on future upgrades and perhaps even new platforms and interface standards?
- What other professional, academic, scientific, commercial, entertainment and governmental fields would benefit from these highly specialized search platforms?
- Would Google, Bing, Yahoo and other commercial search engines benefit from participating with the developers in these projects?
- Would proprietary enterprise search vendors likewise benefit from similar joint ventures with the types of teams described above?
- What entrepreneurial opportunities might arise for vendors, developers, designers and consultants who could provide fuller insight and support for developing customized search platforms?
October 19, 2017 Update: For the latest progress and applications of the Semantic Scholar system, see the latest report in a new post on the Economist.com entitled A Better Way to Search Through Scientific Papers, dated October 19, 2017.
1. These 11 Subway Fold posts cover various AI applications and developments.
2. These seven Subway Fold posts cover a range of IBM Watson applications and markets.
3. A new history of DARPA written by Annie Jacobsen was recently published entitled The Pentagon’s Brain (Little Brown and Company, 2015).
4. See this January 30, 2015 Subway Fold post entitled Timely Resources for Studying and Producing Infographics on this topic.