Getting into XR as a Design Prototyper — An Interview with Suresh Sharma
In a previous article, I wrote about my experience getting into design for augmented reality in an era when there were no commercially available headsets, essentially cobbling different pieces of the spatial computing puzzle together until finally joining Magic Leap in 2016. At the end of that article I promised that I would interview someone newer to the space in a followup piece. As a bonus I’ve also found someone coming at it from a different angle, taking the engineering track, then finding their way into design from there.
I am excited to talk to Suresh Sharma, a friend, former colleague, and prototype engineer, on his transition into design for spatial computing. I met and mentored Suresh at PTC Vuforia, where we both contributed to Expert Capture and Vantage — two augmented reality work instructions applications for the factory assembly line — deployed on HoloLens 2.
Suresh worked closely with our design team, taking concepts envisioned by others, and also some of his own ideas, and implemented them as prototypes that helped us test and prove out some tricky interactions in augmented reality. Since that time Suresh has moved on to work for Qualcomm in Amsterdam as a Senior XR Software Engineer.
Our interview took place intermittently within a Google Doc across continents in early September, 2023.
TS: How did you start off working in technology? What led you to engineering?
SS: I feel like engineering chose me! Math and Computer Science were my favourite subjects in high school, and engineering just felt like the obvious choice for me.
I’ve always had an interest in technology and in keeping up with the latest developments, and VR/AR was on my radar since the Google Cardboard days. I remember using a Cardboard-like contraption to watch Carl Pei launch one of the early OnePlus phones in VR, and learning the absolute basics of VR-specific design using the Google Cardboard Design Lab application.
TS: When did you first take an interest in the design of software? Was there a particular application that peaked your interest and made you want to go more that direction?
SS: During my masters in Aerospace Engineering in TU Delft, I specialized in Human-Machine Interaction. Within the masters, the scope was limited to air-traffic control and remotely piloted aircraft, but the basic principles of HCI are the same across disciplines, so that is my foundation.
In around 2016, I tried Tilt Brush for the first time on a HTC Vive at my university, and that was my first experience of what full 6DOF VR felt like. My mind was blown, and I immediately decided that VR/AR was what I wanted to do going forward! From there, in the following years, I remember playing around with various other experiences that were available at the time, such as the Budget Cuts demo, The Lab by Valve, and this really awesome audio creation experience called SoundStage VR. When I “played” with these and many other VR demos from Steam, what I was really doing was analyzing every design decision, and trying to push the interactions to their limits to see how/if they would break. I learned a lot from these experiences, and they inspired me to start creating my own games and prototypes using Unity.
At my first job developing XR tools for the space engineering industry, I learned that when building enterprise applications, the ease-of-use of XR software was critically important. Unlike early-adopters and enthusiasts who will willingly spend the time required to learn how to use new tech, enterprise clients have a lot of inertia when it comes to adopting new tech into their existing workflow. The easier the tech was to use, and the more familiar it felt, the less hesitation they would have in adopting it. This is where I spent a lot of time deepening my knowledge of software design, UX and interaction design.
TS: When you entered the work world what was going on in the AR/VR/XR space? What excited you about it at the time?
SS: I entered the work world in 2018, and the main devices at the time were the HTC Vive, Oculus Rift and the HoloLens 1. It wasn’t until a year later, with the release of the Oculus Quest, that standalone VR became widely available. What excited me the most back then was the complete novelty of everything — there were no established design best practices or guidelines. In fact, even with the release of multiple standalone VR devices and Apple’s recent announcement, this remain true even today, and it continues to be very exciting! The ability to image fresh interactions using innovative modalities such as hand tracking, eye tracking and BCIs, which could potentially lead to designing something that establishes industry standards.
TS: How did you first break into spatial computing as a professional?
SS: During the last year of my Masters, I got the opportunity to build a multi-sensory VR experience for the International Festival of Technology at TU Delft. A few friends and I built a hot-air balloon ride in VR using Unity, and we connected the Unity game to multiple fans and a heat lamp in the real world using Arduino boards, to add to the feeling of immersion and presence. This was the first time I had the opportunity to build something that would actually be experienced by hundreds of people! (You can check out a video of this experience here: https://youtu.be/c7p_9y7s3TM)
When I started looking for a job towards the end of my masters, I have to say that I was very lucky: I had a masters in Aerospace Engineering, and I had also been teaching myself how to build XR experiences in Unity for the past year — that sounds like a weird combination, right? Who would need such a peculiar mix of skills? Luckily, I found a space engineering company in the Netherlands that was looking for exactly that! Someone with both Aerospace as well as some VR/AR experience — I think even they were a little surprised that they found someone with the skills they were looking for. Over the next 3 years with them, I built many bespoke XR engineering tools for our clients and for internal use.
TS: At what point did you start to embrace design prototyping on spatial computing? How did that change your relationship with those that you worked with in your day to day? What new tasks did you find yourself responsible for?
SS: I began design prototyping pretty early in my career. One significant aspect of my role at the time was advocating for XR technology, which wasn’t always easy because potential clients struggled to see its benefits in their workflow. It was like the Ford scenario where users didn’t know they needed cars instead of faster horses.
For example, we proposed using XR for inspecting stress and thermal analysis results on satellite components instead of traditional PDF reports. Since these components and results are 3D, XR made perfect sense. However, convincing clients was a challenge until they tried it themselves. To make it click, I created numerous small prototypes and proof of concepts, demonstrating the tech’s value firsthand.
In my later role at Vuforia, I originally joined as a software engineer, focusing on building and shipping their enterprise software. However, my interest in design led me to reach out to the designer our team frequently collaborated with, who happened to be you, Tim! Fortunately, we connected well, and my proposal to work closely with both the design and engineering teams received support from both sides. It became clear that having a prototyper who could effectively bridge these two areas was valuable.
My responsibilities included crafting interactive prototypes for the design team to test their concepts, which, in turn, guided our design decisions. This process also helped us identify and address obscure edge cases, preventing potential roadblocks down the line. Leveraging my software engineering background and familiarity with the codebase, my prototypes aided the product manager in estimating time and prioritizing features.
As a prototyper, my role involved collaborating with various teams: design, engineering, as well as product management. This multifaceted collaboration was something I truly enjoyed, and I loved the variety it brought to my work.
TS: It was great working with you and testing the prototypes you made for Vuforia Capture and Vantage applications on HoloLens 2. What is the most effective way for an AR/AR/XR design team to work with prototypers? Can you give an example in your own career where that worked well?
SS: I’m still searching for the most effective way myself, and I don’t feel like I have a definitive answer for this. In my view, a prototyper should function as an integral part of the design team. Being involved in feedback meetings and design reviews is crucial because it provides the necessary context for my design and development work.
What has proven effective in my experience is maintaining short and rapid iteration cycles. For instance, during our collaboration at Vuforia, we would cycle through multiple potential versions of a design in a week. Prototypes played a vital role in helping us identify spatial interactions that worked effectively, while also fitting within the application’s design language.
TS: Where are you interested in going with your career? What excites you most about a spatial computing future?
SS: I love working in the niche between design and engineering, and I hope I continue to do more of that! I would also like my work to include all of our senses, using visuals, audio, haptics, as well as work across both hardware and software.
In the context of the future of spatial computing, what excites me most is its ambient nature, where information becomes accessible when and where you need it. I’m also intrigued by the synergy between XR technologies and our current devices like smartphones and laptops, envisioning XR as a valuable addition to our toolkit rather than a replacement for these devices.
TS: What are some resources for those in Engineering interested in making the transition to AR/VR/XR prototyper?
SS: To get started, I recommend checking out Apple’s design related videos on their developer website (https://developer.apple.com/videos/design), as well as the mixed reality design guidelines by Microsoft (https://learn.microsoft.com/en-us/windows/mixed-reality/design/design) and Meta(https://developer.oculus.com/resources/mr-design-guideline/).
I also highly recommend following a couple of amazing folks in the industry to see what they’re working on and get inspired! Here are some people whose work I find myself going back to consistently:
- Alexandria Heston (x.com/ali_heston). She has written an extremely detailed guide on the design of VR and AR: https://aliheston.gitbook.io/the-design-of-virtual-and-augmented-reality/
- Adam Varga (x.com/dmvrg)
- Mike Alger’s YouTube channel (https://www.youtube.com/@MikeAlger/videos)
- Matthaeus Krenn (https://matthaeuskrenn.com). He has a multi part series on YouTube where he goes into incredible depth on the design of a VR OS: https://www.youtube.com/@MatthaeusKrenn/videos
- Nick Klingelsmith (x.com/koujaku) creating StereoKit (https://stereokit.net)
- Yoon Park (https://www.linkedin.com/in/cre8ivepark/)
- Daniel Beauchamp (x.com/pushmatrix) for some delightful and sometimes mildly-terrifying hand tracking experiments
More information on Suresh Sharma:
Website: sureshsharma.xyz
LinkedIn: https://www.linkedin.com/in/sureshsharmaxyz/
Twitter: x.com/sureshsharmaxyz
