Minting New Big Data Types and Analytics for Investors

Along with the exponential growth of big data in terms of its quantity, myriad of collection points, nearly limitless storage capabilities, and complex analytics, investors are keenly interested in discovering unique advantages from this phenomenon to be applied in the securities markets.* While financial institutions of all types have used sophisticated metrics and predictions to gain tactical advantages in their trading operations for many decades, burgeoning big data methodologies have recently created new opportunities for entrepreneurs to provide the financial services industry with ever more original and arcane forms of predictive analytics.

Investors now have data services available to them offering insights never previously feasible or even imaginable. Yesterday’s (November 21, 2014) edition of The Wall Street Journal carried a fascinating report highlighting three of these operations entitled Startups Tip Investors to Hidden Data Pearls by Bradley Hope. (A subscription to the WSJ Online is required for full access to this report on, but this piece was available here in slightly different version on CBS’s This additional extract page from the article is also available online and contains explanatory graphics of their formats and analyses.

How are these new data points being mined, examined and spun into forecasts? To briefly sum up the work of these startups covered in this article:

  • Orbital Insight analyzes satellite photos of building sites in 30 cities in China, cornfields, and parking lots in order to assess how their capacities might influence the markets in various ways. They are seeking to intuit “early indicators” of trends and influences. Their clients include hedge funds.
  • Dataminr sifts through a half a billion daily tweets in order to spot potential market moving trends ahead of the news services.** The link above to the graphics from the WSJ article contains a very effective infographic on this process.*** The company’s proprietary systems categorize and analyze all tweets in real time, discerns potentially useful patterns, and then distributes the results to their clients.
  • Premise Inc. uses a global system, now in 18 countries, that provides cell phone credits as payments to individuals who monitor the prices of various goods. From this input, the company tracks early inflation rates and other economic data. They believe that their data can differ from official government sources.

I recommend reading this story in full for all of its compelling details.

My follow up questions include:

  • Who watches these watchmen? Will market forces determine which of them are producing valid and actionable collection and analytics or should they somehow be subject to regulatory oversight?
  • Because these data types and analytics are so new, how are these companies and others like them addressing the distinctions between correlation and causation in their reports to their clients? Would it be beneficial for them to form a trade association to address this and other issues that might arise in the future for this nascent industry?
  • Are there entrepreneurial opportunities here for another type of new startups to vet the practices and products of such companies? That is, analysts who produce no new data types themselves, but rather, apply existing and, perhaps develop new, analytical tools for such assessments?
  • What other fields, markets and professions might benefit from this trend to discover and assess new data types in addition to finance?


*    Please see this April 9, 2014 Subway Fold post entitled Roundup of Some Recent Books on Big Data, Analytics and Intelligent Systems.

**   Please see this July 31, 2014 Subway Fold post entitled New Analytical Twitter Traffic Report on US TV Shows During the 2013 – 2014 Season.

***  Please see this January 30, 2015 Subway Fold post entitled Timely Resources for Studying and Producing Infographics.

Twitter Invests $10M in Establishing the Laboratory for Social Machine at MIT

The astronomical diversity of Twitter users and topics never ceases to expand and amaze. Everyone and their neighbor from #anthropologists to #zoologists and countless others post approximately 500 million Tweets each day. This produces a virtual ocean of highly valuable data and accompanying analytics that have found applications in, among a multitude of other areas, e-commerce, marketing, entertainment, government, sports, academia, science, medicine and law. For example, two recent Subway Fold posts here have looked at the mappings of Twitter networks and the analysis of Twitter traffic about TV shows to examine this phenomenon.

Taking this to yet another level of involvement and sophistication was an announcement on October 1, 2014 that was posted on entitled Twitter Gives MIT $10M and Access to the Firehose to Build a Laboratory for Social Machines, by Matthew Ingram. To briefly recap, Twitter is providing funding for a new undertaking at MIT called the Laboratory for Social Machines (LSM). Its mandate is to examine the effects of social media on society, including the creation of new tools (such as pattern recognition and data visualization), and methodologies for doing so. They further intend to create a platform where the findings can be openly discussed and possibly acted upon by the interested parties.

LSM will have access to the entire quantum of Twitter posts going back to the social platform’s launch in 2006. Other planned participants will include journalists, “social groups and movements”. Their website provides more fine-grained details about their objectives, approaches and personnel. I highly recommend clicking through to the LSM site to learn more and get a genuine sense that this could really be something big. As well, their own new Twitter feed is @mitlsm.

Additional coverage of this story can be found here on The Wall Street Journal’s Digits blog and here on the Boston Business Journal’s techflash blog.

What a remarkable and admirable leap forward this is for Twitter and MIT. At its outset, this sounds like a venture that is destined to produce practical and actionable benefits to nterested groups across the real and virtual worlds, not to mention the positive publicity and good will this announcement has already generated.

My own questions include:

  • Will other interested parties be invited to provide funding or is this an exclusive venture between Twitter and MIT?
  • What types of new startups will the work of LSM inspire and support? Will LSM expand itself to become an incubator of some sort?
  • What policies will guide the LSM’s decision-making on the types of studies, tools, movements and so on to pursue? Is establishing an advisory board in their current plans?
  • Will other universities build comparable labs for social media studies?
  • Will professional organizations, trade associations, and other specific interest groups likewise create their own such labs?