AR + #s = $$$: New Processes and Strategies for Extracting Actionable Business Data from Augmented Reality Systems

“Ars Electronica – Light Tank”, image by Uwe Rieger (Denz), Yinan Liu (NZ) (Arcsec Lab) @ St. Mary’s Cathedral

Perhaps taking two disparate assertions, one tacit and one spoken, completely out of their original contexts and re-mixing and re-applying them to a different set of circumstances can be a helpful means to introduce an emerging and potentially prosperous new trend.

First, someone I know has a relatively smart and empathetic dog who will tilt his head from side to side if you ask him (the dog) something that sounds like a question. His owner claims that this is his dog’s way of non-verbally communicating – – to paraphrase (or parabark, maybe) – – something to the effect of “You know, you’re right. I never really thought if it that way”. Second, in an article in the January 4, 2019 edition of The New York Times entitled The Week in Tech: Amazon’s Burning Problems, by David Streitfeld, there is an amusing quote from a writer for WIRED named Craig Mod describing his 2018 Amazon Kindle Oasis as being “about as interactive as a potato”.

So, let’s take some literary license (and, of course, the dog’s license, too), and conflate these two communications in order to paws here to examine the burgeoning commercial a-peel of the rich business data now being generated by augmented reality (AR) systems.

To begin, let’s look no further than the 2019 Consumer Electronics Show (CES) held last month in Las Vegas.  New offerings of AR products and services were all the rage among a number of other cutting-edge technologies and products being displayed, demo-ed and discussed.¹ As demonstrably shown at this massive industry confab, these quickly evolving AR systems that assemble and present a data-infused overlay upon a user’s real-world line of sight, are finding a compelling array of versatile applications in a widening spectrum of industries. So, too, like everything else in today’s hyper-connected world, AR likewise generates waves of data that can be captured, analyzed and leveraged for the benefit and potential profit of many commercial enterprises.

A close and compelling examination of this phenomenon was recently posted in an article entitled Unlocking the Value of Augmented Reality Data, by Joe Biron and Jonathan Lang, on the MIT Sloan Management Review site on December 20, 2018. I highly recommend a click-through and full read if you have an opportunity. I will try to summarize and annotate this piece and, well, augment it with some of my own questions.

[The Subway Fold category of Virtual and Augmented Reality has been tracking a sampling of developments in this area in commerce, academia and the arts for the past several years.]

Image from Pixabay.com

Uncensored Sensors

Prior to the emergence of the Internet of Things (IoT), it was humans who mostly performed the functions of certain specialized sensors in tasks such as detecting environment changes and then transmitting their findings. Currently, as AR systems are increasingly deployed, people will be equipped with phones and headsets, among other devices, embedded with these sensing capabilities. This “provides uncharted opportunities for organizations” to make use of the resulting AR data-enabled analyses to increase their “operational effectiveness” and distinguish the offerings of their goods and services to the consumer public.

AR’s market in 2019 is analogous to where the IoT market was in 2010, garnering significant buzz and demonstrating “early value for new capabilities”. This technology’s capacity to “visualize, instruct, and interact” can become transformative in data usage and analytics. (See Why Every Organization Needs an Augmented Reality Strategy, by Michael E. Porter and James Heppelman, Harvard Business Review, November – December 2017.)

To thereby take advantage of AR, businesses should currently be concentrating on the following questions:

  • How best to plan to optimize and apply AR-generated data?
  • How to create improved “products and processes” based upon AR users’ feedback?

Image from Pixabay.com

AR Systems Generate Expanding Spheres of User Data

Looking again to the past for guidance today, with the introduction of the iPhone and Android phones in 2007 and 2008, these tech industry turning points produced “significant data about how customers engaged with their brand”. This time period further provided engineers with a deeper understanding of user requirements. Next, this inverted the value proposition such that “applications could sense and measure” consumer experiences as they occurred.

Empowered with comparable “sensing capabilities emerging through the IoT”, manufacturers promptly added connectivity, thus generating the emergence of smart, connected products (SCPs). These new devices now comprise much of the IoT. The resulting massive data collection infrastructure and the corresponding data economy have been “disrupting technology laggards ever since”.

Moreover, using “AR-as-a-sensor” for gathering deep quantities of data holds significant potential advantages. Many AR-enabled devices are already embedded with sensing capabilities including “cameras, GPS, Bluetooth, infrared and accelerometers”. More organically, they also unleash human “creativity, intuition and experience” that cannot be otherwise replicated by the current states of hardware and software.²

What can humans with AR-based devices provide to enhance their experiences? New types of data and “behavioral insights” can be harvested from both SCPs and unconnected products. For example, in the case of an unconnected product, a user with a device equipped to operate as a form of AR-as-a-sensor could examine how the product is used and what are the accompanying user preferences for it. For an SCP, the AR-equipped user could examine how usage affects performance and whether the product is adaptable to that particular user’s purposes.

For additionally needed critical context, it is indeed “human interaction” that provides insights into how SCPs and unconnected devices are realistically operating, performing and adapting.

“Rainbow Drops”, Image by Mrs. eNil

Evaluating Potential Business Benefits from AR-Derived Data

This new quantum of AR information further creates a form of feedback loop whereby questions concerning a product’s usage and customization can be assessed. This customer data has become central to “business strategy in the new digital economy”.

In order to more comprehensively understand and apply these AR data resources, a pyramid-shaped model called “DIKW” can be helpful. Its elements include

  • Data
  • Information
  • Knowledge
  • Wisdom

These are deployed in information management operations to process unrefined AR data into “value-rich knowledge and insights”. By then porting the resulting insights into engineering systems, businesses can enhance their “product portfolio, design and features” in previously unseen ways.

AR data troves can also be merged with IoT-generated data from SCPs to support added context and insights. For unconnected devices or digital-only offerings, humans using AR to interact with them can themselves become sensors similarly providing new perspectives on a product’s:

  • Service usage
  • Quality
  • Optimization of the “user experience and value”

“DSC_445”, Image by Frank Cundiff

Preliminary Use Cases

The following are emerging categories and early examples of how companies are capturing and leveraging AR-generated data:

  • Expert Knowledge Transfer: Honeywell is gathering data from experienced employees and then enhancing their collective knowledge to thereafter be transferred to new hires. The company has implemented this by “digitizing knowledge” about their products only made visible through experience. This enables them to better understand their products in entirely new ways. Further details of this initiative is presented on the firm’s website in a feature, photos and a video entitled How Augmented Reality is Revolutionizing Job Training.
  • Voice of the Product: Bicycle manufacturer Cannondale is now shipping their high-end products with an AR phone app to assist owners and bike shop mechanics with details and repairs. This is intended to add a new dimension to bike ownership by joining its physical and digital components. The company can also use this app to collect anonymized data to derive their products’ “voice”. This will consequently provide them with highly informative data on which “features and procedures” are being used the most by cyclists which can then be analyzed to improve their biking experiences. For additional information about their products and the accompanying AR app, see Cannondale Habit Ready to Shred with All-New Proportional Response Design, posted on Bikerumor.com on October 9, 2018. There is also a brief preview of the app on YouTube.
  • Personalized Services: AR is being promoted as “transformative” to online and offline commerce since it enables potential buyers to virtually try something out before they buy it. For instance, Amazon’s new Echo Look permits customers to do this with clothing purchases. (See Amazon’s Echo Look Fashion Camera is Now Available to Everyone in the US, by Chris Welch, posted on TheVerge.com on June 6, 2018.) The company also patented something called “Magic Mirror” in January 2018. When this is combined with Echo Look will point the way towards the next evolution of the functionality of the clothing store dressing room. (See Amazon’s Blended-Reality Mirror Shows You Wearing Virtual Clothes in Virtual Locales, by Alan Boyle, posted on GeekWire.com on January 2, 2018.) The data collected by Echo Look is “being analyzed to create user preference profiles” and, in turn, suggest purchases based upon them. It is reasonably conceivable that combining these two technologies to supplement such personalized clothing recommendations will produce additional AR-based data, elevating “personalized services and experiences” to a heretofore unattained level.³
  • Quality Control: For quite a while, DHL has been a corporate leader in integrating AR technology into its workers’ daily operations. In one instance, the company is using computer vision to perform bar code scanning. They are further using this system to gather and analyze quality assurance data. This enables them to assess how workers’ behavior “may affect order quality and process efficiency”. (See the in-depth report on the company’s website entitled Augmented Reality in Logistics, by Holger Glockner, Kai Jannek, Johannes Mahn and Björn Theis, posted in 2014.)

Image from Pixabay.com

Integrating Strategic Applications of AR-Derived Data

There is clearly a range of meaningful impacts upon business strategies to be conferred by AR-derived data. Besides the four positive examples above, other companies are likewise running comparable projects. However, some of them may likely remain constrained from wider exposure because of “technological or organizational” impediments.

With the emergence of AR-generated data resources, those firms that meaningfully integrate them with other established business data systems such as customer relationship management (CRM) and “digital engagement”, will yield tangible new insights and commercial opportunities. Thus, in order to fully leverage these potential new possibilities, nimble business strategists should establish dedicated multi-departmental teams to pursue these future benefits.

My Questions

  • Because the datastreams from AR are visually based, could this be yet another fertile area to apply machine learning and other aspects of artificial intelligence?
  • What other existing data collection and analysis fields might also potentially benefit from the addition of AR-derived data stream? What about data-driven professional and amateur sports, certain specialties of medical practice such as surgery and radiology, and governmental agencies such as those responsible for the environment and real estate usage?
  • What entrepreneurial opportunities might exist for creating new AR analytical tools, platforms and hardware, as well as integration services with other streams of data to produce original new products and services?
  • What completely new types of career opportunities and job descriptions might be generated by the growth of the AR-as-a-sensor sector of the economy? Should universities consider adding AR data analytics to their curriculum?
  • What data privacy and security issues may emerge here and how might they be different from existing concerns and regulations? How would AR-generated data be treated under the GDPR? Whether and how should people be informed in advance and their consent sought if AR data is being gathered about them?
  • How might AR-generated data affect any or all of the arts and other forms of creative expression?
  • Might some new technical terms of ARt be needed such as “ARformation”, “sensAR” and “stARtegic”?

 


1.  Much of the news and tech media provided extensive coverage of this event. Choosing just one report among many, the January 10, 2019 edition of The New York Times published a roundup and analysis of all of the news and announcements that have occurred in an engaging article with photos entitled CES 2019: It’s the Year of Virtual Assistants and 5G, by Brian X. Chen.

2.   For an alternative perspective on this question see the November 20, 2018 Subway Fold post entitled The Music of the Algorithms: Tune-ing Up Creativity with Artificial Intelligence.

3.  During the 2019 Super Bowl 53 played (or, more accurately, snoozed through), on February 3, 2019, there was an ad for a new product called The Mirror. This is a networked full-size wall mirror where users can do their daily workouts in directly in front of it and receive real-time feedback, performance readings, and communications with other users. From this ad and the company’s website, this device appears to be operating upon a similar concept to Amazon’s whereby users are receiving individualized and immediate feedback.

“I Quant NY” Blog Analyzes Public Data Sets Released by New York City

8025834548_a2eb6f2115_z

Image by Justin Brown

[This post was originally uploaded on October 24, 2014. It has been updated below with new information on February 3, 2015.]

Using large data sets that local government agencies in New York City have made available by virtue of the NYC Open Data program, a visiting college professor at Pratt Institute, statistician and blogger named Ben Wellington, has been taking a close quantitative look at some common aspects of everyday life here in the city. He was a guest on The Brian Lehrer Show on WNYC radio in New York on October 16, 2014 to discuss four of his recent posts on his I Quant NY blog presenting the results of several of his investigations and analyses. The nearly 13-minute podcast entitled We Quant NY: Stories From Data is absolutely fascinating as Wellington describes his subjects, results and supporting methodologies.

(X-ref to this Subway Fold post on April 9, 2014 post, in particular to the fourth book mentioned entitled Smart Cities: Big Data, Civic Hackers, and the Quest for a New Utopia by Anthony M. Townsend about other endeavors like this. As well, an article entitled They’re Tracking When You Turn Off the Lights by Elizabeth Dwoskin was published in The Wall Street Journal on October 20, 2014 [subscription required] about current efforts by researchers in New York and elsewhere to place “municipal sensor networks” around the city to gather and study many other data sets about the how the city operations and its residents. Townsend is also quoted in this story.)

The posts and analytics that Mr. Wellington discussed on the radio and online included:

  • Why it is nearly impossible to purchase or refill a MetroCard to pay your transit fares in such an amount that it will have $0.00 left on it. There always seems to be some small amount left no matter what payment option you choose at the vending machines.This irks many of my fellow New Yorkers.
  • Fire hydrants that generate the most tickets for parking violations.
  • The gender difference among the customer base for the Citi Bike sharing program. That is, Citi Bike riders in midtown Manhattan tend to be more male while riders in Brooklyn tend to be more female. Why is this so?
  • Which building in Manhattan is the farthest from the subway. (In his October 23, 2014 blog post, Mr. Wellington has studied and found the residence in Brooklyn which is the farthest from the subway.)

I believe that Mr. Wellington’s efforts are to be admired and appreciated because his is helping us to learn more about how NYC really operates on a very granular level. This can potentially lead to improvements in municipal services and other areas he has explored on his blog such as affordable housing, restaurant chain cleanliness (based upon the data generated by the NYC’s inspection and letter grade rating system), and the water quality and safety of the local swimming areas. I hope that he continues his efforts and inspires others to follow in this citizen’s approach to using publicly available big data for everyone’s benefit.

February 3, 2015 Update:

How interesting could the subject of laundromats in New York possibly be? As it turns out, these washing/drying/folding establishments generate some very interesting data and analytics about the neighborhoods where they operate. Who knew? Let’s, well, press on and see.

A few weeks ago, after Brian Lehrer had guests on his show to discuss President Obama’s State of the Union Address and then New York Governor Andrew Cuomo’s State of the State Address, he then had a segment of his show where he asked callers about the state of the own streets. This was a truly hyper-local topic about a city with a great diversity neighborhoods across its five boroughs. One of the callers to the show from the Upper West Side of Manhattan called in to say that as a result of ongoing real estate development on her street, all of her local laundromats had gone out of  business.

As it turned out, Ben Wellington of the I Quant New York blog (above), heard this and went to work on an analysis to see what the city-wide data might indicate about this. He then returned as a guest on The Brian Lehrer Show on January 28, 2015, to discuss his findings. The podcast available on wnyc.org is entitled Following Up: Are Laundromats Disappearing? Mr. Wellington’s post on his I Quant NY blog, also posted on January 28th, is entitled Does Gentrification Cause a Reduction in Laundromats? I highly recommend clicking through and checking out both of them as remarkable examples of how a deeper look at some rather mundane urban data can produce such surprising results and insights about New York.

On the podcast, they were also joined by author and photographer Snorri Sturluson who wrote a book entitled Laundromat (PowerHouse Books, 2013), and later on by Brian Wallace who is the president of the Coin Laundry Association, a trade group. Mr. Sturluson’s book is a photo album sampling many of the hundreds of laundromats across the entire city. (All ten of its reviews on Amazon.com are for the full five stars.)

The ensuing discussion began with the fundamental question of whether the increased affluence and real estate development in a neighborhood directly leads to a decline in the number of local laundromats. As it turns out, a more nuanced and complicated relationship emerged from the geocoded data. In Mr. Wellington’s mapping the results indicate (as shown on both the podcast page and his blog post), that population density is more likely to be the main determinant of the concentration of laundromats. Affluence in each neighborhood is also a factor, but it should also be evaluated in conjunction with population density. The mapping also shows that certain neighborhoods in Queens such as Astoria and Jackson Heights, have the highest concentrations of Laundromats.

Callers to show raised other possible consideration such as whether there are higher numbers of recent college grads in an area, the emergence of online services that offer full laundry services including pickup and delivery, and even the social acceptability nowadays of going to a laundromat. Here are my follow-up questions:

  • Is population density in this analysis more particular to New York than other cities or, if similarly mapped elsewhere, would the distribution of its impact and statistically weighting appear to be similar in other comparably large cities?
  • What other types of businesses, government agencies, scientists and universities might be interested in these results and in testing such data in other locations?
  • Are there additional patterns of businesses that cluster around laundromats such as supermarkets or restaurants and, if so, how to whom might these data sets and analytics be useful?
  • Will the eternal mystery of where socks lost in the laundry go to ever be solved?