Mixing modes: UX Design for multimodal & AR

Since continuing my exploration from Alexa and Google Assistant. I’ve been curious about how Digital Assistants and AR devices would converge. One of the most difficult things I faced when understanding this was, how exactly would this be useful to the users?

Multimodal design was first introduced in 2016 and made popular in 2018 with the launch of screen-based voice-first devices. When thinking about AR design as part of a voice-forward design process I make sense to include as another platform for a user to consume a companies services and effectively is just virtualisation of physical devices but learning how to port to these platforms.

The Assistants such as Alexa and Google’s offering has allowed me to focus my ideas around how exactly can these tools be useful for the everyday user. I have at length explored how Assistant embedded devices can be used in different situations such as Shopping, Driving and a user’s morning routine and attached this thinking to major world brands.

Why Mixed Reality, Why now and Why AR?

Taking the time to understand and see the potential to help users solve simple customer service problems was the reason for starting to explore mixed reality and try and design an AR (HoloLens2, Focals by North or Magic Leap Lightwear) version of our multimodal voice-first action for Google Action.

Having very much had AR in the back of my mind for many years ever since the emergence of the category of AR-enable smartphone based apps back in 2010 to the development of VR and First generation AR devices such as HoloLens 1 and experimental Google glasses. I choose as a commercial designer to focus on the tech that would be most relevant to real world users and this was Voice enabled digital Assistants such as Alexa and then Google Assistant.

Now we are starting to see voice enabled projects asking the question, what would it look it if I spoke to my screen with the development of Multimodal voice-first enabled devices. ex. Home Hub. Therefor it is a natural progression to start to once again think about AR and Voice together unlike before were I very much thought of them as separated into two categories. I was recently told its Voice-first not Voice only with this in mind

Stepping into AR with Voice & Multimodal design. Having recently started to understand exactly what it would take to design for multimodal for The Assistant which includes touch points on, AndroidTV 9+, WearOS 2+, AndroidAuto, SmartSpeakers and regular Android for SmartPhones. I wanted to push myself to see what would it look like if all the screen devices were virtual devices. This as nearly always leads back to Amazon’s Alexa platform (for some reason new formal factors seem to appear on this platform first.) and ByNorth’s “Focals” device. This device are a quite normal looking pair of regular glasses with a 300px by 300px projection into the users right eye.

What struck me as interested the API JSON commands don’t support 300x wide, just 480px wide for the Echo Spot so instantly I knew this would involve closer involvement if this would go forward.

AR, Voice and eCommerce

With such numbers stating that there will be around 2 billion AR-enabled devices that also have a Google Assistant loaded on them, I wanted to see if a voice-first project could be further extended to an AR experience. Really push designing for multimodal really meant. I cam at this in a practical. What is the TV or Speak was virtual? Could a user access the news just with a pair of glasses?

Starting with market research I wanted to better understand how the AR market has been developing over the last 8 months. Since this was the last I explored the space.

The rise of VTO’s : Real Function

Apple’s ARKit and Google’s ARCore are powering new ways for retailers and brands to engage with consumers to help them make better, more effective buying decisions. From trying on glasses to seeing what products will look like in your home, virtual try-on’s (VTO’s) are quickly becoming AR’s first killer app. Virtual try-on’s are a new form of augmented reality that put products into context for the buying consumer. As we move to mobile-phone based e-Commerce, retailers, brands and manufacturers have new opportunities to connect with consumers and gain far more information about their buying patterns than print alone could ever do. VTO’s in certain product categories are seeing a direct impact on bottom line sales. Home Decor VTO’s like IKEA’s Place app or Shopify’s 3D platform are already seeing increases in sales and decreases in returns.

Virtual Try-on for luxury goods

Retailers now have the power to understand the customer journey a little more clearly with digital tools such as virtual, augmented and mixed reality (XR). The data captured by VTO’s goes deeper than any other medium to date. From geolocating the consumer to heat maps of product views to cloud maps of their actual space, the retailer has unprecedented access to the consumer information and in turn, ways to make the

“In the future, e-commerce penetration is projected to grow from approximately 10% today to greater than 40% in 2026”

World Economic Forum, Future of Retail 2017

The idea that consumers can now try products in their home or on themselves, goes beyond novelty to become a transformational way they purchase things. From sunglasses, jewelry, watches, makeup and shoes to furniture, housewares and backyard renovations, VTO’s are the best way for consumers to make better buying decisions.

Recently, Facebook partnered with Michael Kors to showcase their new 3D programmatic ad campaigns that use advanced facial tracking augmented reality to let you try on a pair of glasses right from your Facebook feed. Snapchat, the preferred social media platform for under 25-year-olds has also entered the game with AR campaigns with Nike and others.

As you can see there are already tons of great examples of brands utilizing Augmented Reality (AR) for retail and e-commerce. This is only the very beginning of this technology and how it will be used.