Recently I was asked to explore the current state of WebAR for use within the product description page. To see if the ideas withstood any scrutiny. Like most designers, by now we’ve all tried VR devices such as Oculus Rift, HTC Vive or DayDream. We may have even used some AR forced apps on iOS 11+ or Android 8+
So taking the idea that I was given. Simple Product Viewer for a Product such as a pair of Sunglasses or T-Shirt I started to get my head around what exactly is AR for Apps and how does that compare to WebAR and lastly what is the current state of its development. So let’s get into it.
Overview of AR for Apps.
Before jumping into WebAR I wanted to get a good understanding of the different types of AR in general. As with any emerging technology, there are going to be competing standards and methods to achieve the same or similar end result. Augmented reality is the technology that expands our physical world, adding layers of digital information onto it. Unlike Virtual Reality (VR), AR does not create the whole artificial environments to replace real with a virtual one. AR appears in direct view of an existing environment and adds sounds, videos, graphics to it. A view of the physical real-world environment with superimposed computer-generated images, thus changing the perception of reality, is the AR.
The term itself was coined back in 1990, and one of the first commercial uses were in television and military. With the rise of the Internet and smartphones, AR rolled out its second wave and nowadays is mostly related to an interactive concept. 3D models are directly projected onto physical things or fused together in real-time, various augmented reality apps impact our habits, social life, and entertainment industry.
AR apps typically connect digital animation to a special ‘marker’, or with the help of GPS in phones pinpoint the location. Augmentation is happening in real time and within the context of the environment, for example, overlaying scores to a live feed sports events.
Types of Augmented Reality
There are 4 types of augmented reality today:
- Marker-based AR. Some also call it image recognition, as it requires a special visual object and a camera to scan it. It may be anything, from a printed QR code to special signs. The AR device also calculates the position and orientation of a marker to position the content, in some cases. Thus, a marker initiates digital animations for users to view, and so images in a magazine may turn into 3D models.
- Markerless AR. A.k.a. location-based or position-based augmented reality, that utilizes a GPS, a compass, a gyroscope and an accelerometer to provide data based on user’s location. This data then determines what AR content you find or get in a certain area. With the availability of smartphones this type of AR typically produces maps and directions, nearby businesses info. Applications include events and information, business ads pop-ups, navigation support.
- Projection-based AR. Projecting synthetic light to physical surfaces, and in some cases allows to interact with it. These are the holograms we have all seen in sci-fi movies like Star Wars. It detects user interaction with a projection by its alterations.
- Superimposition-based AR. Replaces the original view with an augmented, fully or partially. Object recognition plays a key role, without it the whole concept is simply impossible. We’ve all seen the example of superimposed augmented reality in IKEA Catalog app, that allows users to place virtual items of their furniture catalog in their rooms.
How does Augmented Reality work
What is Augmented Reality for many of us implies a technical side, i.e. how does AR work. For AR a certain range of data (images, animations, videos, 3D models) may be used and people will see the result in both natural and synthetic light. Also, users are aware of being in the real world which is advanced by computer vision, unlike in VR.
AR can be displayed on various devices: screens, glasses, handheld devices, mobile phones, head-mounted displays. It involves technologies like S.L.A.M. (simultaneous localization and mapping), depth tracking (briefly, a sensor data calculating the distance to the objects), and the following components:
- Cameras and sensors. Collecting data about user’s interactions and sending it for processing. Cameras on devices are scanning the surroundings and with this info a device locates physical objects and generates 3D models. It may be special duty cameras, like in Microsoft Hololens, or common smartphone cameras to take pictures/videos.
- Processing. AR devices eventually should act like little computers, something modern smartphones already do. In the same manner, they require a CPU, a GPU, flash memory, RAM, Bluetooth/WiFi, a GPS, etc. to be able to measure speed, angle, direction, orientation in space, and so on.
- Projection. This refers to a miniature projector on AR headsets, which takes data from sensors and projects digital content (result of processing) onto a surface to view. In fact, the use of projections in AR has not been fully invented yet to use it in commercial products or services.
- Reflection. Some AR devices have mirrors to assist human eyes to view virtual images. Some have an “array of small curved mirrors” and some have a double-sided mirror to reflect light to a camera and to a user’s eye. The goal of such reflection paths is to perform a proper image alignment.
So how does this compare to state of WebAR.
WebAR is a new emerging area of web development that uses our mobile devices to provide an augmented reality experience. As the name suggests WebAR uses the same underlying technology that any other website uses, this allows us to create AR experiences that are just as dynamic as a typical website or web app. Using a marker to orientate the scene, it uses existing JavaScript 3D rendering libraries to show a 3D object projected on top of what the mobile device’s camera sees. This produces a simple but very immersive augmented reality experience that is usable by anyone with a smartphone.
Limitations
Rendering detailed 3D objects in realtime at 60FPS is computationally expensive. Although we have better integration with our mobile devices and the browser, utilizing the GPU for hardware acceleration is sometimes not possible (lack of dedicated graphics hardware) or not supported by the browser, hardware and/or OS. Fortunately over the next 6 months this will be less and less of an issue as mobile GPUs become standard on mobile devices.
AR devices like HoloLens use lasers and optical sensors to track our viewpoint of the world around us, it uses this information to adjust the perspective of the 3D object so it looks static in the real world. Currently, our mobile devices don’t have the technology required to make these observations. To get around this we use a printed marker to reference the origin of our AR experience, the orientation of this known marker can then be calculated using the optical camera and then the 3D object can be adjusted as needed. This isn’t ideal because it introduces friction into the user experience. In order to mitigate this, we are working on a machine learning solution to this problem, by using the advert itself as the marker.
What is USDZ Format?
In collaboration with Pixar, Apple introduced usdz, the new open file format Apple says is “optimized for sharing” across apps like Messages, Safari, Mail, Files, and News, while retaining powerful graphics and animation features. Apple also notes that with the usdz format, Quick Look for AR also allows users to place 3D objects into the real world to see how something would work in a space, and lets others see the same thing, in the same place, at the same time.
USD files aren’t new. They are single-object container files designed for 3D imaging, augmented reality, and virtual reality applications. USD is the core of Pixar’s 3D graphics production pipeline, and has been since “A Bug’s Life.” It is now in its fourth iteration and used in every 3D authoring and rendering application Pixar uses and develops.
The core USD file format can be used simultaneously by multiple artists on their own layers, which are combined after each user finishes work, and it provides an audit trail of changes for accountability. However, a single USD file is best for a single object. Multiple USD files can be combined, though, and that’s where the usdz format comes in.
According to Pixar, the file format is a “zero compression” zip file of existing USD files, with a USD file acting as a “scene” for the rest of the contained assets. The combined file can be used by a compatible software suite to create a 3D scene by composing many individual files together into successively larger and larger aggregations.
A usdz file is read-only. Editing the contents require the user to unpack the file, and extract the individual parts of it. It appears that Adobe Creative Cloud will be able to do this for users — and the native support for the format in the nearly ubiquitous suite is a very big deal for wide adoption of the standard going forward.
The file is not allowed to access other files or assets external to it, such as higher resolution assets stored in a different directory. However, if those assets are included in the file, and a device does not have any need for them, they do not impact the performance of the rendering at all.
Can I use USDZ in a commerical project?
Apple said it’s working with a number of high-profile companies on the adoption of USDZ, with the most prominent one announced so far being Adobe. At WWDC Abhay Parasnis, Adobe’s executive vice-president & CTO, took to the stage and announced that the company was bringing native USDZ support to Adobe’s Creative Cloud. This means it will soon be available to use in Photoshop, Dimensions and other Adobe products to create AR content.
“We’re going to bring the power of immersive design to Creative Cloud,” he continued, “with a new set of services and applications, including a brand-new iOS application that is going to let all of you design amazing AR experiences quickly.
“So you will be able to bring in images, videos, text, any object from Creative Cloud, directly into an AR environment. In fact, for the first time, with Creative Cloud and iOS, you will have ‘What You See Is What You Get’ editing in AR. It’s pretty cool.”
When can I use USDZ to create AR content?
ARKit 2 and usdz are part of the developer preview of iOS 12. ARKit 2 and usdz support will be available to the wider public in autumn as part of iOS 12 and macOS 10.14 Mojave. As the new format is designed to work with iOS 12, you’ll need to wait until Apple rolls out the update.
So currently, WebAR isn’t possible yet. But soon. The only way to experiment with WebAR is to join the Apple iOS 12 Beta programme. So this doesn’t lend itself to a commercial project.
