AR + #s = $$$: New Processes and Strategies for Extracting Actionable Business Data from Augmented Reality Systems

“Ars Electronica – Light Tank”, image by Uwe Rieger (Denz), Yinan Liu (NZ) (Arcsec Lab) @ St. Mary’s Cathedral

Perhaps taking two disparate assertions, one tacit and one spoken, completely out of their original contexts and re-mixing and re-applying them to a different set of circumstances can be a helpful means to introduce an emerging and potentially prosperous new trend.

First, someone I know has a relatively smart and empathetic dog who will tilt his head from side to side if you ask him (the dog) something that sounds like a question. His owner claims that this is his dog’s way of non-verbally communicating – – to paraphrase (or parabark, maybe) – – something to the effect of “You know, you’re right. I never really thought if it that way”. Second, in an article in the January 4, 2019 edition of The New York Times entitled The Week in Tech: Amazon’s Burning Problems, by David Streitfeld, there is an amusing quote from a writer for WIRED named Craig Mod describing his 2018 Amazon Kindle Oasis as being “about as interactive as a potato”.

So, let’s take some literary license (and, of course, the dog’s license, too), and conflate these two communications in order to paws here to examine the burgeoning commercial a-peel of the rich business data now being generated by augmented reality (AR) systems.

To begin, let’s look no further than the 2019 Consumer Electronics Show (CES) held last month in Las Vegas.  New offerings of AR products and services were all the rage among a number of other cutting-edge technologies and products being displayed, demo-ed and discussed.¹ As demonstrably shown at this massive industry confab, these quickly evolving AR systems that assemble and present a data-infused overlay upon a user’s real-world line of sight, are finding a compelling array of versatile applications in a widening spectrum of industries. So, too, like everything else in today’s hyper-connected world, AR likewise generates waves of data that can be captured, analyzed and leveraged for the benefit and potential profit of many commercial enterprises.

A close and compelling examination of this phenomenon was recently posted in an article entitled Unlocking the Value of Augmented Reality Data, by Joe Biron and Jonathan Lang, on the MIT Sloan Management Review site on December 20, 2018. I highly recommend a click-through and full read if you have an opportunity. I will try to summarize and annotate this piece and, well, augment it with some of my own questions.

[The Subway Fold category of Virtual and Augmented Reality has been tracking a sampling of developments in this area in commerce, academia and the arts for the past several years.]

Image from Pixabay.com

Uncensored Sensors

Prior to the emergence of the Internet of Things (IoT), it was humans who mostly performed the functions of certain specialized sensors in tasks such as detecting environment changes and then transmitting their findings. Currently, as AR systems are increasingly deployed, people will be equipped with phones and headsets, among other devices, embedded with these sensing capabilities. This “provides uncharted opportunities for organizations” to make use of the resulting AR data-enabled analyses to increase their “operational effectiveness” and distinguish the offerings of their goods and services to the consumer public.

AR’s market in 2019 is analogous to where the IoT market was in 2010, garnering significant buzz and demonstrating “early value for new capabilities”. This technology’s capacity to “visualize, instruct, and interact” can become transformative in data usage and analytics. (See Why Every Organization Needs an Augmented Reality Strategy, by Michael E. Porter and James Heppelman, Harvard Business Review, November – December 2017.)

To thereby take advantage of AR, businesses should currently be concentrating on the following questions:

  • How best to plan to optimize and apply AR-generated data?
  • How to create improved “products and processes” based upon AR users’ feedback?

Image from Pixabay.com

AR Systems Generate Expanding Spheres of User Data

Looking again to the past for guidance today, with the introduction of the iPhone and Android phones in 2007 and 2008, these tech industry turning points produced “significant data about how customers engaged with their brand”. This time period further provided engineers with a deeper understanding of user requirements. Next, this inverted the value proposition such that “applications could sense and measure” consumer experiences as they occurred.

Empowered with comparable “sensing capabilities emerging through the IoT”, manufacturers promptly added connectivity, thus generating the emergence of smart, connected products (SCPs). These new devices now comprise much of the IoT. The resulting massive data collection infrastructure and the corresponding data economy have been “disrupting technology laggards ever since”.

Moreover, using “AR-as-a-sensor” for gathering deep quantities of data holds significant potential advantages. Many AR-enabled devices are already embedded with sensing capabilities including “cameras, GPS, Bluetooth, infrared and accelerometers”. More organically, they also unleash human “creativity, intuition and experience” that cannot be otherwise replicated by the current states of hardware and software.²

What can humans with AR-based devices provide to enhance their experiences? New types of data and “behavioral insights” can be harvested from both SCPs and unconnected products. For example, in the case of an unconnected product, a user with a device equipped to operate as a form of AR-as-a-sensor could examine how the product is used and what are the accompanying user preferences for it. For an SCP, the AR-equipped user could examine how usage affects performance and whether the product is adaptable to that particular user’s purposes.

For additionally needed critical context, it is indeed “human interaction” that provides insights into how SCPs and unconnected devices are realistically operating, performing and adapting.

“Rainbow Drops”, Image by Mrs. eNil

Evaluating Potential Business Benefits from AR-Derived Data

This new quantum of AR information further creates a form of feedback loop whereby questions concerning a product’s usage and customization can be assessed. This customer data has become central to “business strategy in the new digital economy”.

In order to more comprehensively understand and apply these AR data resources, a pyramid-shaped model called “DIKW” can be helpful. Its elements include

  • Data
  • Information
  • Knowledge
  • Wisdom

These are deployed in information management operations to process unrefined AR data into “value-rich knowledge and insights”. By then porting the resulting insights into engineering systems, businesses can enhance their “product portfolio, design and features” in previously unseen ways.

AR data troves can also be merged with IoT-generated data from SCPs to support added context and insights. For unconnected devices or digital-only offerings, humans using AR to interact with them can themselves become sensors similarly providing new perspectives on a product’s:

  • Service usage
  • Quality
  • Optimization of the “user experience and value”

“DSC_445”, Image by Frank Cundiff

Preliminary Use Cases

The following are emerging categories and early examples of how companies are capturing and leveraging AR-generated data:

  • Expert Knowledge Transfer: Honeywell is gathering data from experienced employees and then enhancing their collective knowledge to thereafter be transferred to new hires. The company has implemented this by “digitizing knowledge” about their products only made visible through experience. This enables them to better understand their products in entirely new ways. Further details of this initiative is presented on the firm’s website in a feature, photos and a video entitled How Augmented Reality is Revolutionizing Job Training.
  • Voice of the Product: Bicycle manufacturer Cannondale is now shipping their high-end products with an AR phone app to assist owners and bike shop mechanics with details and repairs. This is intended to add a new dimension to bike ownership by joining its physical and digital components. The company can also use this app to collect anonymized data to derive their products’ “voice”. This will consequently provide them with highly informative data on which “features and procedures” are being used the most by cyclists which can then be analyzed to improve their biking experiences. For additional information about their products and the accompanying AR app, see Cannondale Habit Ready to Shred with All-New Proportional Response Design, posted on Bikerumor.com on October 9, 2018. There is also a brief preview of the app on YouTube.
  • Personalized Services: AR is being promoted as “transformative” to online and offline commerce since it enables potential buyers to virtually try something out before they buy it. For instance, Amazon’s new Echo Look permits customers to do this with clothing purchases. (See Amazon’s Echo Look Fashion Camera is Now Available to Everyone in the US, by Chris Welch, posted on TheVerge.com on June 6, 2018.) The company also patented something called “Magic Mirror” in January 2018. When this is combined with Echo Look will point the way towards the next evolution of the functionality of the clothing store dressing room. (See Amazon’s Blended-Reality Mirror Shows You Wearing Virtual Clothes in Virtual Locales, by Alan Boyle, posted on GeekWire.com on January 2, 2018.) The data collected by Echo Look is “being analyzed to create user preference profiles” and, in turn, suggest purchases based upon them. It is reasonably conceivable that combining these two technologies to supplement such personalized clothing recommendations will produce additional AR-based data, elevating “personalized services and experiences” to a heretofore unattained level.³
  • Quality Control: For quite a while, DHL has been a corporate leader in integrating AR technology into its workers’ daily operations. In one instance, the company is using computer vision to perform bar code scanning. They are further using this system to gather and analyze quality assurance data. This enables them to assess how workers’ behavior “may affect order quality and process efficiency”. (See the in-depth report on the company’s website entitled Augmented Reality in Logistics, by Holger Glockner, Kai Jannek, Johannes Mahn and Björn Theis, posted in 2014.)

Image from Pixabay.com

Integrating Strategic Applications of AR-Derived Data

There is clearly a range of meaningful impacts upon business strategies to be conferred by AR-derived data. Besides the four positive examples above, other companies are likewise running comparable projects. However, some of them may likely remain constrained from wider exposure because of “technological or organizational” impediments.

With the emergence of AR-generated data resources, those firms that meaningfully integrate them with other established business data systems such as customer relationship management (CRM) and “digital engagement”, will yield tangible new insights and commercial opportunities. Thus, in order to fully leverage these potential new possibilities, nimble business strategists should establish dedicated multi-departmental teams to pursue these future benefits.

My Questions

  • Because the datastreams from AR are visually based, could this be yet another fertile area to apply machine learning and other aspects of artificial intelligence?
  • What other existing data collection and analysis fields might also potentially benefit from the addition of AR-derived data stream? What about data-driven professional and amateur sports, certain specialties of medical practice such as surgery and radiology, and governmental agencies such as those responsible for the environment and real estate usage?
  • What entrepreneurial opportunities might exist for creating new AR analytical tools, platforms and hardware, as well as integration services with other streams of data to produce original new products and services?
  • What completely new types of career opportunities and job descriptions might be generated by the growth of the AR-as-a-sensor sector of the economy? Should universities consider adding AR data analytics to their curriculum?
  • What data privacy and security issues may emerge here and how might they be different from existing concerns and regulations? How would AR-generated data be treated under the GDPR? Whether and how should people be informed in advance and their consent sought if AR data is being gathered about them?
  • How might AR-generated data affect any or all of the arts and other forms of creative expression?
  • Might some new technical terms of ARt be needed such as “ARformation”, “sensAR” and “stARtegic”?

 


1.  Much of the news and tech media provided extensive coverage of this event. Choosing just one report among many, the January 10, 2019 edition of The New York Times published a roundup and analysis of all of the news and announcements that have occurred in an engaging article with photos entitled CES 2019: It’s the Year of Virtual Assistants and 5G, by Brian X. Chen.

2.   For an alternative perspective on this question see the November 20, 2018 Subway Fold post entitled The Music of the Algorithms: Tune-ing Up Creativity with Artificial Intelligence.

3.  During the 2019 Super Bowl 53 played (or, more accurately, snoozed through), on February 3, 2019, there was an ad for a new product called The Mirror. This is a networked full-size wall mirror where users can do their daily workouts in directly in front of it and receive real-time feedback, performance readings, and communications with other users. From this ad and the company’s website, this device appears to be operating upon a similar concept to Amazon’s whereby users are receiving individualized and immediate feedback.

Startup is Visualizing and Interpreting Massive Quantities of Daily Online News Content

"News", Image by Mars Hill Church Seattle

“News”, Image by Mars Hill Church Seattle

Just scratching the surface, some recent Subway Fold posts have focused upon sophisticated efforts, scaling from startups to established companies, that analyze and then try to capitalize upon big data trends in finance¹, sports² , cities³, health care* and law**. Now comes a new report on the work of another interesting startup in this sector called Quid. As reported in a most engaging story posted on VentureBeat.com on November 27, 2014 entitled Quid’s Article-analyzing App Can Tell You Many Things — Like Why You Lost the Senate Race in Iowa by Jordan Novet, they are gathering up, indexing and generating interpretive and insightful visualizations for their clients using data drawn from more than a million online articles each day from more than 50,000 sources.

I highly recommend a full read of this story for all of the fascinating details and accompanying screen captures from these apps. As well, I suggest visiting and exploring Quid’s site for a fuller sense of their products, capabilities and clients. I will briefly recap some of the key points from this story. Furthermore, this article provides a timely opportunity to more closely tie together seven related Subway Fold posts.

The first example provided in the story concerns the firm’s production of a rather striking graphic charting the turn in polling numbers for a Democratic candidate running for an open Senate seat in Iowa following a campaign visit from Hillary Clinton. When Senator Clinton spoke in the state in support of the Democratic candidate, she address women’s issue. Based upon Quid’s analysis of the media coverage of this, the visit seemed to have helped the Republican candidate, a woman, more then the Democratic candidate, a man.

Politics aside, Quid’s main objective is to become a leading software firm in supporting corporate strategy for its clients in markets sectors including, among others, technology, finance and government.

Quid’s systems works by scooping up its source materials online and then distilling out specific “people, places, industries and keywords”. In turn, all of the articles are compared and processed against a specific query. In turn, the software creates visualizations where “clusters” and “anomalies” can be further examined. The analytics also assess relative word counts, magnitudes of links shared on social media, and mentions in blog posts and tweets. (X-ref again to this November 22, 2014 post entitled Minting New Big Data Types and Analytics for Investors that covers, among other startups, one called Dataminr that also extensively analyzes trends in Twitter usage and content data.)

The sample screens here demonstrate the app’s deep levels of detail in analytics and visualization. As part of the company’s demo for the author, he provided a query about companies involved in “deep learning”.  (X-ref to this August 14, 2014 Subway Fold post entitled Spotify Enhances Playlist Recommendations Processing with “Deep Learning” Technology concerning how deep learning is being used to, well, tune up Spotify’s music recommendation system.) As this is an area the writer is familiar with, while he did not find any unexpected names in companies provided in the result, he still found this reassuring insofar as this confirmed that he was aware of all of the key participants in this field.

My follow up questions include:

  • Would it be to Quid’s and/or Dataminr’s advantage(s) to cross-license some of their technology and provide supporting expertise for applying and deploying it and, if so, how?
  • Would Quid’s visualizations benefit if they were ported to 3-D virtual environments such as Hyve-3d where users might be able to “walk” through the data? (X-ref to this August 28,2014 Subway Fold post entitled Hyve-3D: A New 3D Immersive and Collaborative Design System.) That is, does elevating these visualizations from 2-D to 3-D add to the accessibility, accuracy and analytics of the results being sought? Would this be a function or multiple functions of the industry, corporate strategy and granularity of the data sets?
  • What other marketplaces, sciences, professions and products might benefit from Quid’s, Dataminr’s and Hyve-3D’s approaches that have not even been considered yet?

________________________________
1.  See this November 22, 2014 post entitled Minting New Big Data Types and Analytics for Investors.

2.  See this October 31, 2014 post entitled New Data Analytics and Video Tools Affecting Defensive Strategies in the NFL and NBA.

3.  See this October 24, 2014 post entitled “I Quant NY” Blog Analyzes Public Data Sets Released by New York City.

*  See this October 3, 2014 post entitled New Startups, Hacks and Conferences Focused Upon Health Data and Analytics .

**  See this August 8, 2014 post entitled New Visualization Service for US Patent and Trademark Data .