New Startup’s Legal Research App is Driven by Watson’s AI Technology

"Supreme Court, 60 Centre Street, Lower Manhattan", Image by Jeffrey Zeldman

[New York] “Supreme Court, 60 Centre Street, Lower Manhattan”, Image by Jeffrey Zeldman

May 9, 2016: An update on this post appears below.


Casey Stengel had a very long, productive and colorful career in professional baseball as a player for five teams and later as a manager for four teams. He was also consistently quotable (although not to the extraordinary extent of his Yankee teammate Yogi Berra). Among the many things Casey said was his frequent use of the imperative “You could look it up”¹.

Transposing this gem of wisdom from baseball to law practice², looking something up has recently taken on an entirely new meaning. According to a fascinating article posted on Wired.com on August 8, 2015 entitled Your Lawyer May Soon Ask for This AI-Powered App for Legal Help by Davey Alba, a startup called ROSS Intelligence has created a unique new system for legal research. I will summarize, annotate and pose a few questions of my own.

One of the founders of ROSS, Jimoh Ovbiagele (@findingjimoh), was influenced by his childhood and adolescent experiences to pursue studying either law or computer science. He chose the latter and eventually ended up working on an artificial intelligence (AI) project at the University of Toronto. It occurred to him then that machine learning (a branch of AI), would be a helpful means to assist lawyers with their daily research requirements.

Mr. Ovbiagele joined with a group of co-founders from diverse fields including “law to computers to neuroscience” in order to launch ROSS Intelligence. The legal research app they have created is built upon the AI capabilities of IBM’s Watson as well as voice recognition. Since June, it has been tested in “small-scale pilot programs inside law firms”.

AI, machine learning, and IBM’s Watson technology have been variously taken up in these nine Subway Fold posts. Among them, the September 1, 2014 post entitled Possible Futures for Artificial Intelligence in Law Practice covered the possible legal applications of IBM’s Watson (prior to the advent of ROSS), and the technology of a startup called Viv Labs.

Essentially, the new ROSS app enables users to ask legal research questions in natural language. (See also the July 31, 2015 Subway Fold post entitled Watson, is That You? Yes, and I’ve Just Demo-ed My Analytics Skills at IBM’s New York Office.) Similar in operation to Apple’s Siri, when a question is verbally posed to ROSS, it searches through its data base of legal documents to provide an answer along with the source documents used to derive it. The reply is also assessed and assigned a “confidence rating”. The app further prompts the user to evaluate the response’s accuracy with an onscreen “thumbs up” or “thumbs down”. The latter will prompt ROSS to produce another result.

Andrew Arruda (@AndrewArruda), another co-founder of ROSS, described the development process as beginning with a “blank slate” version of Watson into which they uploaded “thousands of pages of legal documents”, and trained their system to make use of Watson’s “question-and-answer APIs³. Next, they added machine learning capabilities they called “LegalRank” (a reference to Google’s PageRank algorithm), which, among others things, designates preferential results depending upon the supporting documents’ numbers of citations and the deciding courts’ jurisdiction.

ROSS is currently concentrating on bankruptcy and insolvency issues. Mr. Ovbiagele and Mr. Arruda are sanguine about the possibilities of adding other practice areas to its capabilities. Furthermore, they believe that this would meaningfully reduce the $9.6 billion annually spent on legal research, some of which is presently being outsourced to other countries.

In another recent and unprecedented development, the global law firm Dentons has formed its own incubator for legal technology startups called NextLaw Labs. According to this August 7, 2015 news release on Denton’s website, the first company they have signed up for their portfolio is ROSS Intelligence.

Although it might be too early to exclaim “You could look it up” at this point, my own questions are as follows:

  • What pricing model(s) will ROSS use to determine the cost structure of their service?
  • Will ROSS consider making its app available to public interest attorneys and public defenders who might otherwise not have the resources to pay for access fees?
  • Will ROSS consider making their service available to the local, state and federal courts?
  • Should ROSS make their service available to law schools or might this somehow impair their traditional teaching of the fundamentals of legal research?
  • Will ROSS consider making their service available to non-lawyers in order to assist them in represent themselves on a pro se basis?
  • In addition to ROSS, what other entrepreneurial opportunities exist for other legal startups to deploy Watson technology?

Finally, for an excellent roundup of five recent articles and blog posts about the prospects of Watson for law practice, I highly recommend a click-through to read Five Solid Links to Get Smart on What Watson Means for Legal, by Frank Strong, posted on The Business of Law Blog on August 11, 2015.


May 9, 2016 Update:  The global law firm of Baker & Hostetler, headquartered in Cleveland, Ohio, has become the first US AmLaw 100 firm to announce that it has licensed the ROSS Intelligence’s AI product for its bankruptcy practice. The full details on this were covered in an article posted on May 6, 2016 entitled AI Pioneer ROSS Intelligence Lands Its First Big Law Clients by Susan Beck, on Law.com.

Some follow up questions:

  • Will other large law firms, as well as medium and smaller firms, and in-house corporate departments soon be following this lead?
  • Will they instead wait and see whether this produces tangible results for attorneys and their clients?
  • If so, what would these results look like in terms of the quality of legal services rendered, legal business development, client satisfaction, and/or the incentives for other legal startups to move into the legal AI space?

1.  This was also the title of one of his many biographies,  written by Maury Allen, published Times Books in 1979.

2.  For the best of both worlds, see the legendary law review article entitled The Common Law Origins of the Infield Fly Rule, by William S. Stevens, 123 U. Penn. L. Rev. 1474 (1975).

3For more details about APIs see the July 2, 2015 Subway Fold post entitled The Need for Specialized Application Programming Interfaces for Human Genomics R&D Initiatives

Human Resources Management Meets Big Data in Devising Systems to Identify Star Employees

"2009 Leonid Meteor", Image by Ed Sweeney

“2009 Leonid Meteor”, Image by Ed Sweeney

Have you ever seen a clearly talented colleague at your workplace who was not fully recognized for his or her potential?

Today there is a raft of sophisticated data-driven software products being marketed to Human Resources departments (HR) to assist companies in finding possible star employees. However, some of these systems are not living up to their own, well, potential. Employers are still struggling to identify people on their staffs who have might be likely to excel in their future career paths.

This modern workplace quandary was the subject of a very interesting and informative feature in the June 17, 2015 edition of The Wall Street Journal entitled Are Companies Any Good at Picking Stars?, by Rachel Feintzeig.  I will sum up some of the main points, annotate, and ask some additional questions.

Businesses today have a wealth of data about their employees’ performances and productivity. Nonetheless, identifying who among them have the greatest potential to assume leadership roles in the future is still “more art than science”.  Assessments by humans as well as software algorithms are both still lacking in some respects.

As a result, companies including Nokia, American Express and SAP are turning to new means to measure employee potential. These include new forms of metrics and classifications, as well as games to identify leadership characteristics.

No firm has yet constructed a truly breakthrough HR system to accomplish this. Furthermore, a survey entitled Potential: Who’s Doing What to Identify Their Best? to conducted by Talent Strategy Group LLC indicates, among its other findings, that much of the approximately $70B to $75B US spent on corporate training has been “misspent”.

Tom Rauzi, Dell’s Director of Global Talent, will soon be launching a research project to assess employee data including “education, trajectories and performance” in an effort to identify candidates who might be best qualified to move up in the company.

Generally, when managers have workers with high potential, they have a tendency to choose people “who are like them”.  In another survey, this one by US-based management and advisory company CEB Inc., 25% of 9,500 manager surveyed reported that they “reply on gut instinct” when choosing potential future leaders. This might suggest why some businesses are so challenged in locating “fresh thinkers and diverse hires”.

Christopher Collins, “an associate professor at Cornell University’s School of Industrial and Labor Relations and director of its Center for Advanced Human Resource Studies“, reports that workers who sensed their work is being tracked and evaluated for advancement, often stay with their companies longer and work harder.

Conversely, those workers who are not tracked for future leadership may become resentful. As a result of this, SAP North America ended its high potential categorization.

Carie Davis, who until March 2015 was Coca-Cola’s Director of Innovation and Entrepreneurship, sensed that the company’s high potential program was made up mostly of “Type A employees” with common backgrounds. During some meetings, she found that the discussion ended up being more about “jostling for power” than the intended purpose of innovation.

At a management consulting company called Development Dimensions International Inc., a vice president named Matt Paese reported that companies are now using executive level assessment tools to test thousands of employees throughout their companies. His firm is set to soon start offering a “cheaper, lighter version” of their existing executive-level products for this purpose.

Some HR software vendors are devising their own new tools to illuminate potential. Their algorithms draw from a series of metrics including, among others, an employee’s 401(k) contributions, promotions and network connections within their firms.

For example, a system called UltiPro High Performance Predictor from Ultimate Software Group Inc., measures workers on the probability of their performing well, as distinguished from their potential, into future months. Currently, they are extending their research on “predictors of potential”.

Another suppressant of potential leadership in the workplace, rude and disrespectful behavior by management, was covered in a very insightful opinion piece in the June 25, 2015 edition of The New York Times entitled No Time to Be Nice at Work, by Christine Porath. I highly recommend reading this for its many piercing analytical insights as well as an adjunct to this terrific WSJ article by Ms. Feintzeig. I found that these articles overlapped on some points and can be seen as two sides of the same coin in their effects upon today’s workplaces.

My own questions are as follows:

  • In addition to all of the testing, training, metrics collection and analysis that goes on by HR departments, what if any role does the opinion of an employee’s peers have in spotting potential? While there are many businesses that engage in peer evaluations, I wonder whether on a more informal basis, are co-workers also asked to identify which of their colleagues could be future stars?
  • What are the results of follow-up validation studies in those who were promoted along a path to leadership? While the WSJ article explores the faults in these systems, what about the successes? If John and Mary have been vetted for a leadership track, do they more often than not meet such expectations? Are they more or less inclined to change jobs or departments along the way?
  • As companies, consultants and academics continue to experiment with and fine tune their algorithms, what is the relationship between and among data establishing a correlation as opposed to actual causation in identifying leaders? (This issue has also previously been visited in these five Subway Fold posts.)

Finally, for a hilarious take on a completely unqualified and unmotivated fictional employee failing his way up the corporate ladder, I very highly recommend checking out Season 2 of Silicon Valley on HBO. Here is an interview on Tumblr with the actor Josh Brenner, discussing his role as this character named “Big Head”.