VR Interaction Design

2016. HTC Vive ships with room-scale tracking. Valve Index adds finger tracking. Meta Quest 3 drops controllers entirely. Seven years of evolution - from 'hold the wand' to 'just raise a hand'. But every gesture is backed by engineering: 6DOF IMU, ML keypoint detection, locomotion trade-offs between immersion and physiology. Interaction design in VR is not about polished animations. It is about what makes presence feel real.

• Meta Quest: 20+ million units sold, hand tracking used in Meta Horizon Worlds and VR meetings
• Apple Vision Pro: eye tracking + pinch gesture as the primary interaction - no controllers at all
• VR in medicine: surgical training with haptic feedback on a 7DOF robot (Fundamental Surgery, Osso VR)
• Beat Saber: 10+ million copies sold precisely because there is no locomotion - and no sickness

6DOF Controllers: anatomy of virtual hands

2016. HTC Vive ships with room-scale tracking and two controllers. For the first time, hands exist in VR. Valve Index in 2019 adds finger tracking - each finger tracked independently. Meta Quest 3 in 2023 drops controllers entirely. Seven years of evolution: from 'hold the wand' to 'just raise a hand'. Controllers did not disappear though - each approach has its niche.

**6DOF (six degrees of freedom)** - a controller tracks both position (x, y, z) and orientation (pitch, yaw, roll). Inside: an **IMU** (inertial measurement unit) - gyroscope + accelerometer for orientation, and **optical tracking** for position. Two tracking approaches: **inside-out** (cameras on the headset track the controllers - Meta Quest) and **outside-in** (external base stations - Valve Index Lighthouse). The housing also includes trigger, grip, thumbstick, face buttons, and a **haptic actuator** - an LRA (linear resonant actuator) capable of amplitude and frequency control.

**OpenXR** - open standard from Khronos Group (2019). A single API for all platforms: Meta, Valve, Pico, Microsoft HoloLens. Before OpenXR every vendor had a proprietary SDK - code written for Oculus would not run on Vive. Now a single `xrCreateSession()` runs on any OpenXR-compatible device.

**Unity XR Interaction Toolkit** abstracts the specific controller via `InputActionReference`. Instead of `OVRInput.Get(OVRInput.Button.PrimaryIndexTrigger)`, the code uses `triggerAction.action.ReadValue<float>()` - works on any OpenXR-compatible device without changes.

Hand Tracking: computer vision instead of physical buttons

Hand tracking is more accurate than controllers for hand position, but worse for feel - no tactile feedback, no physical buttons. Meta Quest 3 supports both modes precisely because each has a niche. Medical simulators use controllers with haptics; collaborative spaces use hand tracking. This is not competition - it is complementarity.

The technique: depth cameras on the headset capture the hands, an **ML model** (trained on millions of examples) predicts the positions of **21 skeleton keypoints** in real time. **MediaPipe HandLandmarker** is an open-source Google implementation that runs on a phone CPU. **Meta Quest Hand Tracking 2.0** achieves sub-centimeter accuracy at 60 fps. The primary interaction gesture is the **pinch** (thumb + index finger) - the mouse-click equivalent of hand tracking.

**Hand tracking limitations**: 1) Occlusion - hands behind the back or crossing each other are lost. 2) Additional latency of 5-10 ms vs controllers. 3) No haptics - no sensation of pressing a button. 4) Accuracy drops on fast movements. This is why professional VR surgical simulators with force-feedback use specialised controllers rather than hand tracking.

Locomotion: moving without getting sick

Teleportation in VR is not a 'bad' solution. It is a deliberate trade-off: it eliminates locomotion sickness at the cost of breaking presence. Most successful VR games use teleportation - Lone Echo, Superhot VR, Half-Life: Alyx optionally. Beat Saber solved it differently: no locomotion at all - stand still and swing the sabers. 10+ million copies sold. Sometimes the best solution is to not have the problem.

Core patterns: **Teleportation** - arc pointer selects a destination, instant transfer, snap rotation for turning. Eliminates sickness, sacrifices immersion. **Smooth locomotion** - thumbstick movement as in FPS. Maximum immersion, maximum sickness risk. **Arm-swinger** - swinging arms mimics walking, less sickness than smooth. **Redirected walking** - the system imperceptibly rotates virtual space while walking in a straight line, letting a bounded physical room act as an infinite 'treadmill'.

**Oculus comfort rule**: always offer both options (teleport + smooth locomotion) with user-controlled settings. Comfort vignette - darkening the peripheral field during movement - reduces sickness by 30-40% for most users with smooth locomotion. Snap rotation (45-degree steps) instead of smooth turning is another standard solution.

Cybersickness: physiology of discomfort and engineering solutions

2016. First HTC Vive demos. 30% of users experienced nausea within 5-10 minutes. Cybersickness is not psychological weakness. It is a physiological response to conflict between the vestibular system (the body is not moving) and the visual system (the eyes see motion). The brain interprets this mismatch as poisoning and activates a protective reflex. Quest 3: motion-to-photon latency 12 ms. Valve Index: 11 ms at 144 Hz. Every millisecond is the difference between comfort and nausea.

Four primary sources of sickness: **Vection** - visual sense of movement without actual body displacement. **Latency** - delay over 20 ms between head movement and frame update. **FOV** - peripheral vision is especially sensitive to motion; wide FOV during locomotion amplifies the conflict. **Flicker** - refresh rate below 90 Hz creates flicker perceptible at the periphery. Solutions: static comfort frames (HUD elements anchored to the head), vignetting the FOV during movement, snap rotation, 90+ Hz display, correct IPD calibration.

**IPD calibration**: IPD (Inter-Pupillary Distance) averages ~63 mm but ranges from 54-74 mm across the population. Incorrect IPD causes double vision, headache, and amplified sickness - even at correct framerate and latency. Meta Quest 3 has mechanical IPD adjustment in three positions. Valve Index offers continuous adjustment. Calibration before extended sessions is essential.

Key ideas

• **6DOF controller** = IMU (orientation) + optical tracking (position) + haptics. **OpenXR** is the single API for all platforms.
• **Hand tracking** is more accurate for position but loses to controllers on haptics and occlusion reliability. Meta Quest 3 supports both modes.
• **Locomotion** is a trade-off: teleportation eliminates sickness at the cost of immersion; smooth locomotion maximises presence at the cost of comfort. Beat Saber solved it by removing locomotion entirely.
• **Cybersickness** is visual-vestibular mismatch. Latency threshold: 20 ms. Solutions: vignette, snap rotation, 90+ Hz display. User adaptation is not a design solution.

Related topics

VR interaction design builds on rendering, computer vision, and earlier AR/VR lessons.

• Introduction to XR: AR, VR, MR — Core XR concepts are the foundation for interaction design
• AR Tracking — Tracking technologies (inside-out, outside-in) are used in VR controllers too
• Spatial Computing — Spatial UI patterns apply directly to VR interaction
• VR Rendering — Latency and framerate directly drive cybersickness
• Computer Vision — Hand tracking is built on CV: keypoint detection, MediaPipe, depth cameras
• Computer Graphics — Post-processing effects (vignette, comfort frames) are part of the graphics pipeline

Вопросы для размышления

• Apple Vision Pro removed physical controllers entirely, making eye tracking + pinch the primary input. Which categories of applications benefit from this approach and which lose out compared to Quest controllers?
• Redirected walking imperceptibly rotates virtual space while the user walks in a straight line. What is the perceptual limit of 'unnoticeable' rotation, and how does this constraint affect the size of virtual spaces that can be explored in a 4x4 m room?
• Beat Saber solved locomotion sickness by removing locomotion entirely. Which game genres genuinely require locomotion - and are there alternative design solutions for them, or is comfort always a compromise?

Связанные уроки

• cg-01