Seeing the world: why 3D and VR present challenges, and opportunities, for user interfaces
We don’t think about user interfaces when they’re done well. Games have many ways to deliver information to the player, and the best designed games have interfaces that got lost inside the game, and allow you to gather intelligence without having to search the screen with your eyes. After playing certain games on a 3D monitor and thinking about the work that will need to be done to get games playing on the Oculus Rift, however, it has become clear that more work will need to be done to deliver that same information to the player in 3D games, or games that support head-tracking. “The primary metaphor to consider for stereo UI is to envision the monitor or TV as being like a window to another world. For most elements in heads-up displays, you can have them exist as if they’re painted ‘on’ that window, and have no issues,” David Crooks, a UI and systems programmer for Bioware, told the Penny Arcade Report. “That kind of setup is relatively easy to set up, because the game can draw the same UI elements in the same locations for the views drawn for each eye.” You run into problems dealing with icons in the game world that have to exist within that world, in a place beyond the window on 3D display. Even something as simple as an aiming reticle for a gun becomes problematic. “Simply drawing these on the window won’t work, because our eyes can’t focus on things that are at two separate depths: it’s like trying to look down a gun’s sights with both eyes open, or like holding your hand in front of you at arm’s length and then simultaneously trying to focus on your hand and something else across the room. Again, with both eyes open,” Crooks explained. “Specific work needs to be done for those elements to project them into the world, such as putting a laser sight on a gun, or completely re-working how they work behind the scenes, so that they can behave slightly differently in each eye to ensure that they’re in the right position.” Doom 3 has been updated with a laser site so the 3D doesn’t give you a headache, for instance, and there are other tricks that can be used to help dealing with 2D interfaces in games viewed on a 3D monitor. “For example, the HUD can be made much more translucent when in stereo mode, to improve the visibility of the world, but because of how our brain can filter out pieces of information that are at different depths, this doesn’t reduce the readability of the HUD that much,” Crooks said. When you begin to think about the game world being viewed with a head-mounted display like the Rift, however, things get very tricky, very quickly.
Virtual reality is a large design challenge
“There is one main problem that Rift brings to the table with UI design, and that is the total absence of a frame for the UI to sit in: if a monoscopic view is like looking at painting, and stereoscopic rendering is like looking through a window, then VR is like smashing the glass on the window and sticking your head outside,” Crooks said. He also stressed that he has never used a Rift himself, so this is all theoretical. “Many UI elements will simply fall apart without that frame in place. At best, you lose a logical point for some HUD elements to be ‘docked’ to, such as putting a health display in the top-center of the screen, or an ammo counter in the bottom-right, and end up just having them float somewhere in the space in front of the viewer.” Since the Rift offers such a wide field of view, there is nothing as simple as an “edge” to the screen to use when orienting the interface. It will just float out there, somewhere between your eyes and the world of the game.“Basically, every HUD element that previously existed along a screen edge, or used a screen edge as a part of some transition or animation, will have to have adequate form and function, even when that edge is so far away that it is at the periphery of the user’s eye,” Crooks explained. “The only alternative is for the designer to create some artificial frame to work in, such as a helmet or visor, but that can often be counter to the immersive, and wide-open experience that VR strives for.” This is where it gets fun, the solution is to think about how the information would be given to the player in reality. Since there is no screen to attach virtual displays, you need to design for the world itself. “Any informational element that exists in the world, such as an ammo counter on top of a machine gun, light from a sword pointing to a location, a speedometer in the dashboard of a car, or a health meter on a player’s back, doesn’t require much, if any, extra work to function properly in a stereoscopic, or VR, gaming environment, and will almost always look and feel better than any other alternative,” Crooks said. If you can just look down to your speedometer inside a car, you don’t need to show that information as a floating, distracting dial hovering in front of your face. In fact, it was announced the free-to-play Mech game Hawken will launch with support for the Rift, and will be offering an updated version of the cockpit for players using the headset. “With Unreal we can dynamically change the point-of-view and make sure that the latency is as low as possible for the viewers to eliminate motion sickness. We’re going to build out a cockpit that players can buy so they can see the inside of the Mech with Oculus for a more immersive gameplay experience,” Mark Long, CEO of Meteor Entertainment, told Forbes. “This is a subset of something we’re going to be bringing to the PC game. It’s free-to-play, so we want people to be able to customize the game the way they want. In addition to the Mech interior, there may even be a bobblehead on the dash.” I've ordered my own Rift developer kit and I'll be reporting on the demos and game modifications that are released before the consumer version, so expect some thoughts on how well Hawken is translated into virtual reality. It may be more work than expected to bring games into “real” 3D environments, but the possibilities for interfaces that make sense as part of the world and increase immersion are more than worth the worry. We can expect some interesting changes to user interfaces when more developers begin to experiment with the Rift; as of this writing, over 5,000 of the developer kits have been ordered.