Text Analysis Systems Mine Workplace Emails to Measure Staff Sentiments

Image from Pixabay.com

Have you ever been employed in a genuinely cooperative and productive environment where you looked forward each day to making your contribution to the enterprise and assisting your colleagues? Conversely, have you ever worked in a highly stressful and unsupportive atmosphere where you dreading going back there nearly every day?  Or perhaps you have found in your career that your jobs and employers were somewhere in the mid-range of this spectrum of office cultures.

For all of these good, bad or indifferent workplaces, a key question is whether any of the actions of management to engage the staff and listen to their concerns ever resulted in improved working conditions and higher levels of job satisfaction?

The answer is most often “yes”. Just having a say in, and some sense of control over, our jobs and workflows can indeed have a demonstrable impact on morale, camaraderie and the bottom line. As posited in the Hawthorne Effect, also termed the “Observer Effect”, this was first discovered during studies in the 1920’s and 1930’s when the management of a factory made improvements to the lighting and work schedules. In turn, worker satisfaction and productivity temporarily increased. This was not so much because there was more light, but rather, that the workers sensed that management was paying attention to, and then acting upon, their concerns. The workers perceived they were no longer just cogs in a machine.

Perhaps, too, the Hawthorne Effect is in some ways the workplace equivalent of the Heisenberg’s Uncertainty Principle in physics. To vastly oversimplify this slippery concept, the mere act of observing a subatomic particle can change its position.¹

Giving the processes of observation, analysis and change at the enterprise level a modern (but non-quantum) spin, is a fascinating new article in the September 2018 issue of The Atlantic entitled What Your Boss Could Learn by Reading the Whole Company’s Emails, by Frank Partnoy.  I highly recommend a click-through and full read if you have an opportunity. I will summarize and annotate it, and then, considering my own thorough lack of understanding of the basics of y=f(x), pose some of my own physics-free questions.

“Engagement” of Enron’s Emails

By Enron [Public domain], via Wikimedia Commons

Andrew Fastow was the Chief Financial Officer of Enron when the company infamously collapsed into bankruptcy in December 2001. Criminal charges were brought against some of the corporate officers, including Fastow, who went to prison for six years as a result.

After he had served his sentence he became a public speaker about his experience. At one of his presentations in Amsterdam in 2016, two men from the audience approached him. They were from KeenCorp, whose business is data analytics. Specifically, their clients hire them to analyze the email “word patterns and their context” of their employees. This is done in an effort to quantify and measure the degree of the staff’s “engagement”. The resulting numerical rating is higher when they feel more “positive and engaged”, while lower when they are unhappier and less “engaged”.

The KeenCorp representatives explained to Fastow that they had applied their software to the email archives of 150 Enron executives in an effort to determine “how key moments in the company’s tumultuous collapse” would be assessed and a rated by their software. (See also the February 26, 2016 Subway Fold post entitled The Predictive Benefits of Analyzing Employees’ Communications Networks, covering, among other things, a similar analysis of Enron’s emails.)

KeenCorp’s software found the lowest engagement score when Enron filed for bankruptcy. However, the index also took a steep dive two years earlier. This was puzzling since the news about the Enron scandal was not yet public. So, they asked Fastow if he could recall “anything unusual happening at Enron on June 28, 1999”.

Sentimental Journal

Milky Way in Mauritius, Image by Jarkko J

Today the text analytics business, like the work done by KeenCorp, is thriving. It has been long-established as the processing behind email spam filters. Now it is finding other applications including monitoring corporate reputations on social media and other sites.²

The finance industry is another growth sector, as investment banks and hedge funds scan a wide variety of information sources to locate “slight changes in language” that may point towards pending increases or decreases in share prices. Financial research providers are using artificial intelligence to mine “insights” from their own selections of news and analytical sources.

But is this technology effective?

In a paper entitled Lazy Prices, by Lauren Cohen (Harvard Business School and NBER), Christopher Malloy (Harvard Business School and NBER), and Quoc Nguyen (University of Illinois at Chicago), in a draft dated February 22, 2018, these researchers found that the share price of company, in this case NetApp in their 2010 annual report, measurably went down after the firm “subtly changes” its reporting “descriptions of certain risks”. Algorithms can detect such changes more quickly and effectively than humans. The company subsequently clarified in its 2011 annual report their “failure to comply” with reporting requirements in 2010. A highly skilled stock analyst “might have missed that phrase”, but once again its was captured by “researcher’s algorithms”.

In the hands of a “skeptical investor”, this information might well have resulted in them questioning the differences in the 2010 and 2011 annual reports and, in turn, saved him or her a great deal of money. This detection was an early signal of a looming decline in NetApp’s stock. Half a year after the 2011 report’s publication, it was reported that the Syrian government has bought the company and “used that equipment to spy on its citizen”, causing further declines.

Now text analytics is being deployed at a new target: The composition of employees’ communications. Although it has been found that workers have no expectations of privacy in their workplaces, some companies remain reluctant to do so because of privacy concerns. Thus, companies are finding it more challenging to resist the “urge to mine employee information”, especially as text analysis systems continue to improve.

Among the evolving enterprise applications are the human resources departments in assessing overall employee morale. For example, Vibe is such an app that scans through communications on Slack, a widely used enterprise platform. Vibe’s algorithm, in real-time reporting, measures the positive and negative emotions of a work team.

Finding Context

“Microscope”, image by Ryan Adams

Returning to KeenCorp, can their product actually detect any wrongdoing by applying text analysis? While they did not initially see it, the company’s system had identified a significant “inflection point” in Enron’s history on the June 28, 1999 date in question. Fastow said that was the day the board had discussed a plan called “LJM”, involving a group of questionable transactions that would mask the company’s badly under-performing assets while improving its financials. Eventually, LJM added to Enron’s demise. At that time, however, Fastow said that everyone at the company, including employees and board members, was reluctant to challenge this dubious plan.

KeenCorp currently has 15 employees and six key clients. Fastow is also one of their consultants and advisors. He also invested in the company when he saw their algorithm highlight Enron’s employees’ concerns about the LJM plan. He hopes to raise potential clients’ awareness of this product to help them avoid similar situations.

The company includes heat maps as part of its tool set to generate real-time visualizations of employee engagement. These can assist companies in “identifying problems in the workplace”. In effect, it generates a warning (maybe a warming, too), that may help to identify significant concerns. As well, it can assist companies with compliance of government rules and regulations. Yet the system “is only as good as the people using it”, and someone must step forward and take action when the heat map highlights an emerging problem.

Analyzing employees’ communications also presents the need for applying a cost/benefit analysis of privacy considerations. In certain industries such as finance, employees are well aware that their communications are being monitored and analyzed, while in other businesses this can be seen “as intrusive if not downright Big Brotherly”. Moreover, managers “have the most to fear” from text analysis systems. For instance, it can be used to assess sentiment when someone new is hired or given a promotion. Thus, companies will need to find a balance between the uses of this data and the inherent privacy concerns about its collection.

In addressing privacy concerns about data collection, KeenCorp does not “collect, store on report” info about individual employees. All individually identifying personal info is scrubbed away.

Text analysis is still in its early stages. There is no certainty yet that it may not register a false positive reading and that it will capture all emerging potential threats. Nonetheless it is expected to continue to expand and find new fields for application. Experts predict that among these new areas will be corporate legal, compliance and regulatory operations. Other possibilities include protecting against possible liabilities for “allegations of visa, fraud and harassment”.

The key takeaway from the current state of this technology is to ascertain the truth about employees’ sentiments not by snooping, but rather, “by examining how they are saying it”.

My Questions

  • “Message In a Bottle”, Image from Pixabay.com

    Should text analysis data be factored into annual reviews of officers and/or board members? If so, how can this be done and what relative weight should it be given?

  • Should employees at any or all levels and departments be given access to text analysis data? How might this potentially impact their work satisfaction and productivity?
  • Is there a direct, casual or insignificant relationship between employee sentiment data and up and/or down movements in market value? If so, how can companies elevate text analysis systems to higher uses?
  • How can text analysis be used for executive training and development? Might it also add a new dimension to case studies in business schools?
  • What does this data look like in either or both of short-term and long-term time series visualizations? Are there any additional insights to be gained by processing the heat maps into animations to show how their shape and momentum are changing over time?

 


1.  See also the May 20, 2015 Subway Fold post entitled A Legal Thriller Meets Quantum Physics: A Book Review of “Superposition” for the application of this science in a hard rocking sci-fi novel.

2These 10 Subway Fold posts cover other measurements of social media analytics, some including other applications of text analytics.

The Successful Collaboration of Intuit’s In-House Legal Team and Data Scientists

"Data Represented in an Interactive 3D Form", Image by Idaho National Laboratory

“Data Represented in an Interactive 3D Form”, Image by Idaho National Laboratory

Intuit’s in-house legal team has recently undertaken a significant and successful collaborative effort with the company’s data scientists. While this initiative got off to an uneasy start, this joining (and perhaps somewhat of a joinder, too), of two seemingly disparate departments has gone on to produce some very positive results.

Bill Loconzolo, the Intuit’s VP of Data Engineering and Analytics, and Laura Fennel, the Chief Counsel and Head of the Legal, Data, Compliance and Policy, tell this instructive story and provide four highly valuable object lessons in an article entitled Data Scientists and Lawyers: A Marriage Made in Silicon Valley, posted on July 2, 2015 on VentureBeat.com. I will sum up, annotate, and pose a few questions of my own requiring neither a law degree nor advanced R programming skills to be considered.

Mr. Loconzolo and Ms. Fennel initially recognized there might be differences between their company’s data scientists and the in-house Legal Department because the former are dedicated to innovation with “sensitive customer data”, while the latter are largely risk averse. Nonetheless, when these fundamentally different mindsets were placed into a situation where they were “forced to collaborate”, this enabled the potential for both groups to grow.¹

Under the best of circumstances, they sought to assemble “dynamic teams that drive results” that they could not have achieved on their own. They proceeded to do this in the expectation that the results would generate “a much smarter use of big data”. This turned out to be remarkably true for the company.

Currently, the Data Engineering and Analytics group reports to the Legal Department. At first, the data group wanted to move quickly in order to leverage the company’s data from a base of 50 million customers. At the same time, the Legal Department was concerned because of this data’s high sensitivity and potential for damage through possible “mistake or misuse”. ² Both groups wanted to reconcile this situation where the data could be put to its most productive uses while simultaneously ensuring that it would be adequately protected.

Despite outside skepticism, this new arrangement eventually succeeded and the two teams “grew together to become one”. The four key lessons that Mr. Loconzolo and Ms. Fennel learned and share in their article for teaming up corporate “odd couples” include:

  • “Shared Outcome”:  A shared vision of success held both groups together. As well, a series of Data Stewardship Principles were written for both groups to abide. Chief among them was that the data belonged to the customers.
  • “Shared Accountability”:  The entire integrated team, Legal plus Data, were jointly and equally responsible for their outcomes, including successes and failures, of their work. This resulted in “barriers” being removed and “conflict” being transformed into “teamwork”.
  • “Healthy Tension Builds Trust”: While both groups did not always agree, trust between them was established so that all perspectives “could be heard” and goals were common to everyone.
  • “A Learning Curve”: Both groups have learned much from each other that has improved their work. The legal team is now using the data team’s “rapid experimentation innovation techniques” while the data team has accepted “a more rigorous partnership mindset” regarding continually learning from others.

The authors believe that bringing together such different groups can be made to work and, once established, “the possibilities are endless”.

I say bravo to both of them for succeeding in their efforts, and generously and eloquently sharing their wisdom and insights online.

My own questions are as follows:

  • What are the differences in lawyers’ concerns and the data scientists’ concerns about the distinctions between correlation and causation in their conclusions and actions? (Similar issues have been previously raised in these six Subway Fold posts.)
  • Is the Legal Department collecting and analyzing its own operation big data? If so, for what overall purposes? Are the data scientists correspondingly seeing new points of view, analytical methods and insights that are possibly helpful to their own projects?
  • What metrics and benchmarks are used by each department jointly and separately to evaluate the successes and failures of their collaboration with each other? Similarly, what, if any, considerations of their collaboration are used in the annual employee review process?

1.  Big data in law practice has been covered from a variety of perspectives in many of the 20 previous Subway Fold posts in the Law Practice and Legal Education category.)

2.  For the very latest comprehensive report on data collection and consumers, see Americans’ Views About Data Collection and Security, by Mary Madden and Lee Rainie, published May 20, 2015, by the Pew Research Center for Internet Science and Tech.