The Augmented Consumer: How AR and AI are Changing Customer Experience and E-Commerce


Cathy Hackl, Futurist and Co-Author of ‘Marketing New Realities: An Introduction to VR & AR Marketing, Branding & Communications’, writes about AR and explores how AI is shaping up the e-commerce and retail industry with delivering customer experience like never before.

E-commerce is hotter than ever and shows no signs of slowing down. The flip side of this success is that everyone is selling their wares online, often with the same e-commerce set up. You know the drill: search, browse, images, descriptions, shopping cart, rinse, repeat, and often return. Moreover, when the shopping experience is this generic, why not just do all your shopping online?

Millennials and Gen-Z want more from the shopping experience, and they often use technology to expand the way they interface with their world. They want customization, interactivity, and an enhanced experience. Augmented reality and Artificial Intelligence can provide these things on an eCommerce site. However, before we get into how these technologies are being used by brands today, let’s clarify what we are talking about as there is plenty of confusion about what they are.

The Technologies

Augmented Reality or “AR” is digital information overlaid over the real world. AR is not a new technology. One can say humans have been trying to augment their realities ever since they started drawing on caves or even trace AR back to 1862 when English scientist John Henry Pepper used light and mirrors to project a ghost onto a theater stage in England/ This is commonly referred to as Pepper’s Ghost. Think about Disney’s Haunted Mansion and how you might’ve first experienced an augmented reality.

If not on the Haunted Mansion ride, then think about 1998, when the NFL broadcast the first football game that showed a yellow first-down line on the field. The only difference then and today is that we can do much more elaborate AR using the smartphones we carry in our pockets. Also, what starts with the rectangles in our hands and our walls (mobile phones & TVs) will eventually become head-mounted displays.

On the other hand, the word Artificial Intelligence or “AI” seems to be everywhere in the news. For many, it is an umbrella term for a series of technologies that appear intelligent in nature or perform functions normally done by humans. AI’s roots go back to early work with machine translation in the 1960s but had a rocky history of unfulfilled promises right thru the late 1990s. But AI as we know it today, building on these decades of research, and driven by much more powerful computers, has now become one of the most important emerging technologies to impact marketing (and humanity as a whole). It has given us self-driving cars, voice-driven digital assistants, and visual search.

In some ways, the consumer of today has been augmented. They have at their hands mobile phones that allow them to scan items in the real world and immediately be able to find out where to buy an item or identify an unknown product. AR and computer vision have given consumers the upper hand, and marketing pros need to implement martech strategies that can guide the brands they represent as they integrate AR and AI into their business practices.

How  AR Can Make an Impact On Your Marketing

Furniture companies like Ikea and Wayfair use Augmented Reality to enhance the experience of using their eCommerce app. An item found in their catalog can be shown as a detailed 3D model and using AR can be placed in the real world so you can see how it will fit. This is not only a fun interactive way to shop, but it has cut down on costly returns. Companies like Warby Parker use AR to show what glasses will look like on your face, with WatchBox doing the same for showing wristwatches on your wrist. AR enhances the customer experience by showing what a product will look like before you buy it. The term used to describe some of these examples is aptly called the virtual try-on feature.

From a social media marketing perspective, Snap and Facebook (and Instagram) have been clear leaders in the integration of AR into marketing campaigns.

Carolina Arguelles, head of AR monetization product strategy at Snap, explains how Snap views AR. “The camera is where your audience is, and it is the starting point for the new OS, and OS that is more than just taking photos,” Arguelles said. “The camera has gone through some fundamental shifts from 2008 to 2018 that makes now the time for AR to actually take off, take off at scale, and take off in a way that marketers can take advantage of,”

For Chris Barbour, Head of Augmented Reality (AR) Camera Partnerships at Facebook, AR is playing an increasingly important role in the marketing mix, and that every brand can benefit from crafting and shaping a point of view for where AR fits in its marketing strategy.

“We have seen success with brands across the gamut, ranging from entertainment (Justice League, Jurassic World) to product try-on (Sephora, Marc Jacobs). These examples were effective because they were focused on awareness and brand building or easing the path to purchase, two marketing objectives that AR has proven impactful,” said Barbour.

Barbour also thinks brands should focus on the use cases that drive the most value in AR. They should identify what AR can offer that’s not possible with other channels, and then build the AR effects or experiences that accomplish that singular purpose. He also recommends giving the consumer agency. “Make people the focus and give them control. Brands should be creating an opportunity for people to tell their own unique stories using the AR effects they’ve created. Keep in mind the person is the main character, not the brand,” he suggested.

Social AR/VR influencer Navah Berg agrees, “AR filters can significantly boost engagement with your audience and can be tailored to any specific topic or event you want to promote. AR enhances any static image and creativity is king when devising ways to differentiate and set yourself apart to give your audience what they want, engagement with a twist of magical fun. Remember, the shiny objects need to be measured, so make sure to A/B test filters versus other content and collect data to support and learn from social media campaigns.”

Mike Boland, Chief Analyst at ARtillery Intelligence, foresees that as time goes on and advertisers gain greater levels of comfort and skill with AR, new commerce functionalities will allow them to take a “full-funnel” approach.

AI and the Personalized Experience

Olay has well over 200 skincare products available, how do you choose what’s right for you? You need an expert, and Olay has built one into its eCommerce site. Olay Skin Advisor uses artificial intelligence, a selfie, and a few questions to make product recommendations for your skin type and concerns with friction-free “add to cart” buttons. Makeup sites like You Cam Makeup and Sephora use a combination of AI and AR to help select makeup for you and show you what it will look like on your face. In these cases, AI enhances the customer experience by acting as a recommendation engine saving the customer time and making the shopping experience more fun.

Pinterest is one of the social media companies that has been leading the way when it comes to computer vision. I recently moderated a panel with Chuck Rosenberg, Pinterest’s Head of Computer Vision and it was interesting to hear the continued advancements the company is doing in visual search and how it’s slowly experimenting with video. During the panel, Rosenberg mentioned how Pinterest’s visual search team continues to work on their visual discovery engine to identify and return visual components from a broader scene to recommend ideas like an outfit or living room style, and differentiate and personalize across queries.

Pinterest recently announced “Complete the Look,” which leverages rich scene context to recommend visually compatible results in Fashion and Home Decor Pins. According to the company, “Complete the Look takes context like an outfit, body type, season, indoors vs. outdoors, various pieces of furniture, and the overall aesthetics of a room, to power taste-based recommendations across visual search technology.” This is an excellent example of how computer vision can truly personalize the shopping and sharing experience online and offline.

The company has a team of AI experts working at their Pinterest Labs seeking to tackle the most challenging problems in Machine Learning and Artificial Intelligence

Future Consumers

As millennials continue to age, Gen Z continues to grow, and Boomers become more technically savvy, the augmentation of the consumer will only continue. The tech points to a future where consumers know precisely what they want and when they want it, and brands need to work even harder to meet their demands.

This article first appeared in

Seeking to build and grow your brand using the force of consumer insight, strategic foresight, creative disruption and technology prowess? Talk to us at +9714 3867728 or mail: or visit

About Author

Comments are closed.