My Master’s thesis explored the trade-off between usability and fidelity in modern VR systems within the context of user training. The core question: how realistic does a VR training environment need to be before the added complexity starts hindering the learning process?

Introduction

To achieve the highest possible fidelity available at the time, I used the Oculus Rift DK2 for visual immersion and a Creative DepthSense camera (a depth-sensing camera) for hand and finger tracking. This combination allowed users to interact with virtual objects using their actual hands, creating a more natural training environment than traditional controller-based systems.

Challenges

Finger Tracking: This consumed the majority of development time. Achieving reliable and consistent finger tracking required finding the optimal camera positioning and implementing sophisticated filtering algorithms to smooth out the inherently noisy tracking data. Small jitters in hand position could make interaction feel unnatural and break immersion.

Natural Interaction Design: Developing an intuitive method for picking up and attaching Lego pieces was critical. Since the long-term goal was VR training research, the interaction techniques needed to be as natural as possible - users shouldn’t need any training to use the training system itself. The interaction model had to feel instinctive.

Calibration System: I modeled the virtual environment to match the real-world lab space where the system would be used, minimizing the disconnect between physical and virtual spaces. However, I also wanted the flexibility to swap in different virtual environments for different training scenarios. This required designing a robust calibration system that could adapt to both precise real-world matching and more generic virtual settings.

Real-World Modeling: Full disclosure: I’m not a trained 3D artist. Most of my modeling skills came from curiosity and self-teaching, so the assets lack the polished post-processing of professional work. However, I focused heavily on getting the scale absolutely right. Proper scale adds tremendous fidelity to a VR experience with relatively little effort - even without perfect lighting and textures, correctly sized objects significantly enhance the sense of presence and realism.

Performance Analysis and Monitoring: Since the ultimate goal was VR training research, data collection was paramount. I built a comprehensive monitoring system that tracked user actions within the 3D environment and assessed their performance against training objectives. This data would later inform research on VR training effectiveness.

It’s worth noting that this was a work in progress - many assets still needed modeling and numerous bugs required fixing. What you see in the video represents approximately 30% of the total work completed during my thesis period.

The Journey

As a solo project, this thesis required extensive research and prototyping across multiple domains I’d never worked in before. The development process was the classic R&D experience: trial and error, countless failures, sleep-deprived debugging sessions, and brain-melting bugs that seemed impossible to solve until they suddenly weren’t.

Software packages used:

  • Unity 3D
  • Intel Perceptual Computing SDK 2013
  • Blender
  • Photoshop

This project was my first real dive into Research and Development, and despite the challenges (or perhaps because of them), I had an absolute blast. There’s something uniquely satisfying about building systems that push the boundaries of what’s possible, even when most of the work ends up on the cutting room floor.