Why XR?

Written on February 15th, 2021 by Orbital

Why is XR valuable, useful or important to us?

If I had to pick one reason I’d argue that the key benefits are primarily social. There’s a way that we as humans interact in the real world that has been hard to capture on traditional online services such as Facebook or Twitter. In the next few years it feels like we’ll come to see “social 3d” as a normal part of our online experience.

But it’s worth drilling down a bit and picking apart specific aspects of this - when I say social what do I really mean? It’s also worth noting that each of these topics could very well be an entire business by itself. It’s clearly beyond the scope of this project to address them all, but in our own work we do want to keep these fundamental values in mind.

There are kinds of work that you can do with other people that you cannot do by yourself. We can barnstorm hard problems together, using many minds to do heavy lifting And computers give us a kind of “physics” that can be more flexible; we can organize vast amounts of work in ways that are impossible in the real world. Wikipedia is a good example of a monumental work created by many minds working together. It’s arguable if it could have been done at all without computers. But, sometimes when we try to build virtual collaboration environments we throw out some important social and physical cues - ways that humans traditionally interact. What we want is the benefits of computing in general but as well a possibility of real world interactions that we are already used to.

To do this well we need to understand some of the unspoken behaviors that people engage in - our communication is much more than just words. We’re not just minds floating in vats exchanging pure information. Many of use are used to largely “disembodied” representations of presence. Facebook, Twitter, Slack, Discord and even Zoom to some degree are disembodied. But part of what makes us human is more than words, or emoji and is more than even our faces: it is our whole bodies.

If you’ve ever spent some time in Mozilla Hubs, or in any social 3D virtual world you’ll quickly see that there are all kinds of hints and cues that tell you what’s going on in a conversation that are actually hard to convey otherwise - that don’t show up in slack or zoom at all. These are patterns we take for granted in the real world. People can virtually appear to stand back a little bit - their avatar physically moves backwards - and this simple virtual motion suggests that they’re interested, but not necessarily wanting to speak. Alternatively somebody can move to the center, and this indicates excitement - without verbally interrupting.

In Social Virtual Reality people can even kind of straddle multiple conversations. You can virtually be between two clusters - and virtually listen to both a little bit. This can signal to others that you’re trying to decide which cluster to join, or that both clusters are interesting. Beyond this there are simple cues about entering and leaving, and an observer can quickly get a total membership count of each cluster of speakers. Virtually there is all kinds of body language - which way one is facing, the avatar one has chosen, and an infinity of other rich visual cues. While we sometimes might like to imagine the future of conversations as being disembodied and abstract it turns out that we all have a lot of wetware that is dedicated towards understanding more than just words. While for people who are blind, or hard of hearing, some of this may need to be shifted to other mediums - these cues at least exist and there is a possibility of saying “Janet is standing on the left a little back from the group and is wearing a sculpture of Rodin’s Thinker as a costume”.

This isn’t to say that faces themselves are not hugely significant(such as we have in Zoom), or that words are not significant - but rather that tone, nuance, context can be conveyed with full spectrum communication. Even hand gestures can play a crucial part in conversation.

2. Stylization of Identity

At the same time, there are some benefits to sometimes be somewhat disembodied. It can be quite fun have an idealized avatar. This can reduce social pressures and can create a sense of safety, and even fun. We all tend to adopt and wear identity like fashion. In Virtual Reality, and possibly even Augmented Reality you could choose to be more flamboyant if you wish, or you can wear signifiers that indicate your interests, capabilities, reputation in a group. I do imagine that we will see more plumage, more self-expression and that this will come to be an accepted part of our social representations. I do believe that virtual fashion or virtual identity will come to be seen as one of the fundamental values of 3d self-expression and that we’ll come to take it for granted.

3. The power of shared experiences

We are, both as individuals and as organizations and companies, always looking for ways to create a sense of a team, or to have people bond around shared experience.

We can talk about a sense of being in a room with people, and we can talk about ideas around how we choose to represent ourselves, but also it is worth focusing on how curation of the space itself, by a skilled orator, creator or Dungeon Master even, can help powerful shared experiences that help build durable relationships between people.

A central argument for immersive 3D, VR and AR is an idea of presence and embodiment. There have been several studies of VR learning tha have argued that VR in particular can create a sense of presence that may contribute to learning. There certainly can be something visceral about being in VR and feeling like you are standing on top of virtual Mount Everest (as opposed to merely reading about it). When you’re in a VR world with friends - then something magic can happen. At best it is as if you all share some kind of new powerful experience together.

I think we’ve all been to corporate team building events with escape-rooms, or dinner parties, or other attempts to build a sense of shared social connection. And I think we can all remember powerful friendships we made simply by having powerful shared experiences with other people. I don’t presume to know the mechanics here, but it seems like (to me at least) a sense a shared journey, experience or adventure can be a chance to rely on the people you are around, to learn what they’re good at, and to build closer relationships. There are also some theories of friendship formation that suggest that children in particular tend to form friendships “over” objects rather than directly - and this might work for people who are strangers to each other as well. This can be especially hard to do when people are work-remote or when the costs of doing so are prohibitive - and so shared VR experiences can be a way to produce some of these effects.

4. Digital Twinning and visual representation of complicated artifacts

We all have complicated personal histories that are hard to communicate. We all have specific important events, important objects, artifacts that have meaning. The same is true for any organization. When I worked at Xerox Parc it was a delight to visit their small museum which showcased some of their earlier work - it showed what they thought was important, what they were proud of, and it helped me understand the organization and its values. If we could let other people wander through a kind of mind palace of what we thought was important, or perhaps a kind of museuem, it could be a great way to communicate what we value.

But beyond is, many of us, both individuals and groups, have places or objects that are meaningful. An individual may have places on a map that they visited that they think are special. A company may have different offices, and there may be small stores and services around those offices that they feel are worth sharing. There may even be factory floors, and digital models of those factory floors. Earlier in my career I worked on simulating industrial operator-in-the-loop training rigs for large machines that were too dangerous for novices to use in the real world. This also is a form of digital twin; in terms of providing a training framework.

More commonly a person may simply have lots of friends, emails, archives, photographs, journals, bookmarks, books, essays, software. We’re each surrounded by representations of things that we value. Any group, non-profit, for-profit, information organization is the same. If there are ways to surface some of these artifacts, or even just the major zones, then it may be easier for a newcomer to learn about that party or entity.

A good example of all this is imagine if you’re a volunteer firefighter and you show up at the firestation for your first day of work. If the entire firestation was decorated with augmented reality markup - showing where various machines were, how they worked, what the procedures and process was - it might be a way to at least get a feeling for the scope of competencies that you needed to develop. And this is a good example of where the future web can go - far beyond being a 2d interface to text - and to becoming a situated 3d interface that lets us communicate “in place” contextually; without switching mental modes.

5. Doing Work in a way that is visible to others

I hope that someday we could all collaborate on big software projects visually with other people together. Software programming is unusual that it is both simple but also opaque. An architect works on a visual representation of their final building in a way that even novices can understand. And software programming is not unlike plumbing, it simply isn’t that hard - but because of the grammars we use as programmers it can seem incomprehensible. The same is almost certainly true of many other industries. Our pets must watch us staring and typing into our computers and think we lead exceedingly drearly lives! If there is some way to both make what we do visible to others, then it may be easier to recruit collaboration. What are normally single person projects could become barn-storming efforts. We already see many tools such as Figma empowering social collaboration. But perhaps even tasks like programming can become prosocial - where your work is represented visually and people can see you wiring modules together. This can also help apprentices get up to speed, and make work simpler, more social and more fun.

6. Durable

Lastly, what feels important to me is a sense of durable persistence of 3d objects. So many of our conversations feel ephemeral. You can be on zoom or in a slack and after the conversation is over anything that was posted there is gone. If conversations are engaged in virtual rooms, perhaps those rooms can persist the artifacts, or perhaps people can collectively build together in 3D, grooming and organizing a shared mind palace that is easy to return to, remember and understand.