Startup is Visualizing and Interpreting Massive Quantities of Daily Online News Content

"News", Image by Mars Hill Church Seattle

“News”, Image by Mars Hill Church Seattle

Just scratching the surface, some recent Subway Fold posts have focused upon sophisticated efforts, scaling from startups to established companies, that analyze and then try to capitalize upon big data trends in finance¹, sports² , cities³, health care* and law**. Now comes a new report on the work of another interesting startup in this sector called Quid. As reported in a most engaging story posted on on November 27, 2014 entitled Quid’s Article-analyzing App Can Tell You Many Things — Like Why You Lost the Senate Race in Iowa by Jordan Novet, they are gathering up, indexing and generating interpretive and insightful visualizations for their clients using data drawn from more than a million online articles each day from more than 50,000 sources.

I highly recommend a full read of this story for all of the fascinating details and accompanying screen captures from these apps. As well, I suggest visiting and exploring Quid’s site for a fuller sense of their products, capabilities and clients. I will briefly recap some of the key points from this story. Furthermore, this article provides a timely opportunity to more closely tie together seven related Subway Fold posts.

The first example provided in the story concerns the firm’s production of a rather striking graphic charting the turn in polling numbers for a Democratic candidate running for an open Senate seat in Iowa following a campaign visit from Hillary Clinton. When Senator Clinton spoke in the state in support of the Democratic candidate, she address women’s issue. Based upon Quid’s analysis of the media coverage of this, the visit seemed to have helped the Republican candidate, a woman, more then the Democratic candidate, a man.

Politics aside, Quid’s main objective is to become a leading software firm in supporting corporate strategy for its clients in markets sectors including, among others, technology, finance and government.

Quid’s systems works by scooping up its source materials online and then distilling out specific “people, places, industries and keywords”. In turn, all of the articles are compared and processed against a specific query. In turn, the software creates visualizations where “clusters” and “anomalies” can be further examined. The analytics also assess relative word counts, magnitudes of links shared on social media, and mentions in blog posts and tweets. (X-ref again to this November 22, 2014 post entitled Minting New Big Data Types and Analytics for Investors that covers, among other startups, one called Dataminr that also extensively analyzes trends in Twitter usage and content data.)

The sample screens here demonstrate the app’s deep levels of detail in analytics and visualization. As part of the company’s demo for the author, he provided a query about companies involved in “deep learning”.  (X-ref to this August 14, 2014 Subway Fold post entitled Spotify Enhances Playlist Recommendations Processing with “Deep Learning” Technology concerning how deep learning is being used to, well, tune up Spotify’s music recommendation system.) As this is an area the writer is familiar with, while he did not find any unexpected names in companies provided in the result, he still found this reassuring insofar as this confirmed that he was aware of all of the key participants in this field.

My follow up questions include:

  • Would it be to Quid’s and/or Dataminr’s advantage(s) to cross-license some of their technology and provide supporting expertise for applying and deploying it and, if so, how?
  • Would Quid’s visualizations benefit if they were ported to 3-D virtual environments such as Hyve-3d where users might be able to “walk” through the data? (X-ref to this August 28,2014 Subway Fold post entitled Hyve-3D: A New 3D Immersive and Collaborative Design System.) That is, does elevating these visualizations from 2-D to 3-D add to the accessibility, accuracy and analytics of the results being sought? Would this be a function or multiple functions of the industry, corporate strategy and granularity of the data sets?
  • What other marketplaces, sciences, professions and products might benefit from Quid’s, Dataminr’s and Hyve-3D’s approaches that have not even been considered yet?

1.  See this November 22, 2014 post entitled Minting New Big Data Types and Analytics for Investors.

2.  See this October 31, 2014 post entitled New Data Analytics and Video Tools Affecting Defensive Strategies in the NFL and NBA.

3.  See this October 24, 2014 post entitled “I Quant NY” Blog Analyzes Public Data Sets Released by New York City.

*  See this October 3, 2014 post entitled New Startups, Hacks and Conferences Focused Upon Health Data and Analytics .

**  See this August 8, 2014 post entitled New Visualization Service for US Patent and Trademark Data .

Robots and Diamonds and Drones, Aha! Innovations on the Horizon for 2015

"DSCOVR Liftoff", Image by NASA Goddard Photo and Video

“DSCOVR Liftoff”, Image by NASA Goddard Photo and Video

Towards the end of each year, much of the tech and business media traditionally make predictions about which technologies to watch during the following year. These prognostications are always of interest to, among others, people working in technology, the sciences, startups, academia and financial services. However, many of these predictions often do not live up to their hype or expectations.

Nonetheless, on November 20, 2014, correspondent Sam Grobart hosted a 45-minute presentation on Bloomberg Television entitled The Year Ahead: Innovation about ten technologies ranging from food to virtual reality that conveyed a remarkable sense of excitement and anticipation from start to finish. Any of these ten could reasonably be expected to find their markets and break out, but only time will tell. All of them are offering highly imaginative and innovative systems that could potentially and significantly impact domestic and global populations and markets.

Be assured that this is not a puff piece on these innovators and their work. Rather, this is a declaration of the work being done by very smart and resourceful people striving to make a difference in their fields. I highly recommend viewing this in its entirety.

This intriguing peek over the horizon featured the following:

  • Soft Robotics Inc. is working on the design and materials for robots with soft exoskeletons for applications in rehabilitative medicine, space exploration and athletic clothing.
  • Thalmic Labs has created an input device called the Myo Armband for computers and other electronics that consists of a band of sensors worn on the user’s forearm that interprets different physical movements as gestural input.
  • Hyve 3D – As discussed in a Subway Fold post on August 28, 2014 entitled Hyve-3D: A New 3D Immersive and Collaborative Design System is a very agile and extensible virtual development platform. This program takes you inside of it with its director, Tomas Dorta, at the University of Montreal and further reports on how Hollywood is now using the system to create virtual storyboards and other projects are in for building construction.
  • AeroFarms is fabricating and implementing “vertical farms” using aeroponics to grow crops, faster and more energy efficiently than under traditional farming, in a system using mists of water rather than planting in soil.
  • Beyond Meat is creating imitation chicken and beef products using plant components to achieve comparable tastes and food textures.
  • Akahn Technologies is growing industrial diamonds and integrating them into computer chips in place of heat sinks thus significantly reducing heat generation and enabling much slimmer computing devices to be manufactured.
  • Suneris is working on a product called “Vetigel” that fully stops major bleeding from wounds and burns within a matter of seconds.
  • The Zephyr from Airbus a solar powered, geostationary hybrid drone and satellite being built and tested to provide broadband network access to underserved territories.
  • Groundwater Replenishment System by the Orange County Water System is purifying waste water and groundwater back into drinkable water in drought-stricken Southern California.
  • Jaunt is developing an entirely new platform and 360 degree camera to create fully immersive virtual reality movies to be viewed using the versatile new Oculus Rift headset.*

Mr. Grobart’s reporting and the production values on this show are absolutely top flight in every journalistic respect. None of the ten endeavors are over- or undersold, but rather, objectively formatted and presented. The production is internally consistent in its narratives about each one and sequenced in a logical order. Thus, the micro view of each startup or project and the macro view of these innovators innovating is done in a highly professional and accessible manner. IMHO, these are all great stories well told.**

In a world filled with so much troubling news each day, it was incredibly refreshing and inspiring to watch this program and try to assess the potential these undertakings.

*  See the cover story from the May 2014 issue of WIRED entitled The Inside Story of Oculus Rift and How Virtual Reality Became Reality by Peter Rubin for comprehensive coverage of this technology and company.

**  X-ref to this November 6, 2014 Subway Fold post entitled Say, Did You Hear the Story About the Science and Benefits of Being an Effective Storyteller?

Music Visualizations and Visualizations About Music

There is a rare neurological condition called synesthesia where a person’s senses become linked in unusual ways. People affected by this may perceive, among other things, numbers as having specific colors or shapes as having tastes. Such people are termed “synesthetes” and this condition is not always seen as a detriment by them. This phenomenon has been studied and written about extensively in scientific literature as well as used as a plot element in fiction writing.

A number of inventive and strikingly beautiful visualizations of music have appeared online that can best be described as “sythesthetic”. That is, they are video creations to show what the music looks like to the graphics artists who have created them. Here is a sampling of some:

There are also two new tools available to provide innovative perspectives on:

All of these links produce big musical fun and I highly recommend clicking through to entertain and enlighten you eyes and ears.

Mapping All the Stars in the Milky Way and All the Devices in the Web Way

This week, has posted two articles that present extraordinary visualizations of all the known stars in our own celestial home – – no, not of Hollywood – – but rather, The Milky Way, while the other is of our own virtual world representing by all devices connected to the Web. I think that viewing them together makes for a very thought-provoking juxtaposition of the celestial and terrestrial/virtual worlds, and side-by-side comparison of their individual density. Moreover, they each display their striking vastness and beauty.

First, in an article entitled Incredible New Milky Way Map Is The Most Detailed Survey Of Our Stellar Home Ever Created, by Jessica Orwig, posted on September 16, 2014, we are presented with a “fish-eye mosaic” of the 219 million stars! in The Milky Way that have been cataloged to date. The report provides the technical on how a groups of scientists at University of Hertfordshire in the UK. The report characterizes this project as being an application of big data technology by the school’s astronomers. IMHO, the team members who worked on this are stars in their own right.

Second, is a report entitled This World Map Shows Every Device Connected To The Internet by Pamela Engel, posted on September 14, 2014. John Matherly at Shodan (which desscribes itself on its home page as ” Shodan is the World’s First Search Engine for Internet-Connected Devices”). The article provides the steps taken to generate this incredible visualization. What it very limns is the geographical inequality of available online access. For example, the US and Europe have far more dense levels of connectivity than some other countries and even entire continents. As well, there is an inconsistent relationship between certain areas’ population density and the cumulative numbers of web-connected devices.

I very highly recommend either opening these features and their accompanying graphics in two separate browser tabs and then toggling between them or alternatively opening two browsers and re-sizing them so both images can be seen simultaneously on the same screen. I believe both of these visualizations are testaments to the ever-increasing imagination of scientists who can construct plan and construct them.

I wonder though what, if any, are the possible commonalities of the structures, densities, patterns of change, and mapping processes of the Milky Way and the Net? Do the astronomers and the Net’s cartographers have anything procedurally and/or scientifically to learn from each other’s efforts?

Conflating the messages and information of both of these graphics further made me think that they might present an updated interpretation of the classic line spoken by the visiting alien to Gort, his servant robot, of “Klaatu barada nikto” in the original 1951 version of the sci-fi classic The Day the Earth Stood Still. According to this article on Wikipedia, the author of the screenplay, Edmund North, is quoted as saying this meant “There’s hope for earth, if the scientists can be reached”. As I see it, by providing this more grand perspective, the alien visitor was trying to teach the people of Earth that their planet is part of a much larger universe and they must be responsible for their actions and consequences affecting the larger spheres. Here too, by virtue of the, well, astronomical effort and originality that went into these new maps, perhaps the scientists responsible for them, at least to some degree, appreciate that message.

New Visual Perspectives on Tweeting, Movies and Life

Remember when birds were the only ones still tweeting? Well, not to twitter* away too much time on this point, but I highly recommend checking out a new visualization of nightingale’s and a canary’s vocalizations were posted on in a brief post on August 21, 2014 entitled 3D Bird Sound Visualization is Electrifying. (The direct link to this appears in the text.) The title understates the 2:30 video created by a multimedia artist named Andy Thomas. IMHO, this is an extraordinarily beautiful visualization that transposes the mellifluous sounds of these birds singing into an animation of how this “look” to the artist. Furthermore, if you find this as entertaining and imaginative as I did, I further recommend a click-through and full perusal of the artist’s blog called Nature Remixed displaying a series of galleries of his beautiful “motion art” and other still-frame graphics. Be sure to scroll down on the home page for the technical details of how he make this creations.

(Looking at these pages also reminded me of the breathtakingly imaginative artwork of Roger Dean, particularly on the Rock Posters page of his website.**)

Fortunately, published another highly original visualization just two days later on August 23rd, entitled The Colors of Motion is an Interactive Visualization of Movie Color. (Likewise, the direct link is in the text.) Here designer Charlie Clark has devised a means to display the color palettes of 27 well known films from the last 30 years or so. Clicking on any of the movie titles takes you to a screen where the “average color” of hundred of frames from the film have been analyzed by a methods developed by Clark. For instance, it always seemed to me that in The Matrix, there seemed to be a greenish tint to many scenes while in Avatar it appeared to be more of a blue-ish tint to much of it. Both films are part of this project. Thus, as you click through the sample frames you will see these hues change in some fascinating ways. Simply this has to be seen and explored first hand to fully appreciate just how clever this visualization is in examining this particular aesthetic element of many iconic films. Please also try a click-through to his full Charlie Clark website to enjoy a deep and wide display of his immense artistic talent.

I very much hope that the artist will apply this analysis to more films from the past, present and then in the future. I wonder how the vibrantly colored palette of this summer smash hit Guardians of the Galaxy would be parsed by this.

Finally, for another dramatically different visualization that is artistically sophisticated in its presentation, not about art per se, but rather, about the author’s life, I suggest checking out an article that was posted on on August 27, 2014, entitled An Infographic Genius Plots Out Another Insanely Detailed Year of His Life by Joseph Flaherty. This concerns, as their creator has termed them “Annual Reports” in the form of multi-dimensional inforgraphic displays of nearly 100,000 data points recorded detail of designer Nicholas Felton’s life during 2013. He has been generating these productions since 2005 (all available on his personal website which is linked to within this story’s text). To say that this is very granular does not even begin to describe it. The details he has charted about seemingly everything he did during this year appears to go to an nearly quantum level never seen anywhere else before. With the rapid advancements in all manner of electronically recording personal data as well as the tools for analyzing and visualizing it, Felton talks about his incorporation of these means within the article.

Other people have been engaged in similar activities in recent years which has come to be known as “lifelogging“. (See also A Modest Proposal: Everyday Lifelogging by Charles Q. Choi, posted November 30, 2011 on Taking this sort of activity another step, hop, crawl and leap  forward also appeared in an article in the September 3, 2014 edition of The New York Times entitled Here, Ansel! Sit, Avedon! Apparently some of the participants have pursued this endeavor with dogged determination and tried to hog the spotlight, while others who were more reticent had to be brought out of their shell to get them to engage.
* Yes, small “t”.
** Yes, many from Yes.

Hyve-3D: A New 3D Immersive and Collaborative Design System

Collaborative tools have certainly come a long way since drawing on restaurant napkins and the backs of envelopes. Sure, these methods are still used today, but for a quantum leap into the future of idea sharing I highly recommend a article that appeared on an August 10, 2014 entitled Spectacular 3-D Sketching System Revolutionizes Design Interaction and Collaboration. This report covers an extraordinary new system called Hyve -3D that was developed at the University of Montreal. On the day this story was published, this system was presented at the SIGGRAPH 2014 Conference.

As described in this report, Hyve-3D (an acronym for “Hybrid virtual Environment 3D”), is a full immersive space where collaborators can create, shape and test new designs for products such as cars and many others. Through a series of input tools such as tablets, designs can be manipulated in a multitude of ways from this highly in-depth environment.

The U of Montreal is current pursuing ways to commercialize this technology, promoting its cost-effectiveness and relative simplicty in comparison to other systems like this currently on the market.

The eye-popping (albeit 2D), accompanying photos show how this works in car design. Many other field are anticipated such as, among others, architecture, medicine and game design. The Hyve-3D website contains more photos of the system in action. Many other field are anticipated such as, among others, architecture, medicine and game design.

This looks to me like the Holodeck some to life where the possibilities can barely be imagined yet. As I recall from Star Trek: Next Gen, there were several episodes where Geordi use the Holodeck to design and configure technological solutions to problems confronting The Enterprise.

I suspect that 3D design collaboration will find many unanticipated uses. Moreover, when combined with other leading edge technologies and materials science, designers will only be limited by their imaginations. For instance, if a 3D printing system were added to Hyve-3D, medical devices could be customized for individual patients needs. The current state of research in this nascent area was covered in a remarkable story also posted on on August 21, 2014 entitled Researchers Use 3D Printers to Create Custom Medical Implants.

Compressing 2,600 Years of Historial Migrations into a 5.5 Minute Video

Nearly redefining the whole notion of, well, compression technology, a groups of scientists in the US and Europe have created and uploaded onto YouTube a 5 minute and 36 second video of the advance of western civilization. Really, it does. They have accomplished this by virtue of a fascinating animation that displays and connects the birthplaces (in blue) and the corresponding death sites (in red) of 100,000 leading figures in history. It begins with Leonardo DiVinci’s birth in 1452 and continues to modern times. The narrator describes the resulting migration patterns of these people across Europe, then to the US, and then onto the other continents. The video and a brief summary of it appeared on on July 31, 2014 in a post entitled Watch 2,600 Years of Culture Spread Across the World in 5 Minutes.

Highly recommend if you have some time to view it and consider its implications. As well, the comments posted on this page pose a series of very interesting questions about the inclusion or exclusion of certain civilizations, the overall effects of what the video intended to show, and the visualization techniques that were used. As to the latter of these, imho, this represents a highly original way to depict human and cultural migration patterns. As well, I hope that it inspires other historians to create other new types of visualizations and animations to represent other developmental patterns across time.