For my Virtual Reality course during my Master’s program, our team was tasked with creating a VR system for constructing and deconstructing Lego structures. Working as a team of three developers and two artists, we set out to create an experience that would feel as natural as playing with physical Lego bricks.

Introduction

The project leveraged the capabilities of UT Dallas’s Motion Capture Lab, which was equipped with Vicon cameras for high-precision tracking. We developed the system using FIVE-UDE, a custom Unity build specifically designed for applications in the University of Texas at Dallas (UTD) motion capture lab. The setup tracked an Oculus Rift DK1 for visual immersion and two Wii controllers that served as the user’s hands, providing both positional tracking and input control.

Given access to such high-precision motion tracking technology, we saw an opportunity to push beyond typical VR interaction techniques. Our goal was ambitious: create an interaction system that could authentically replicate the tactile experience of snapping Lego bricks together and pulling them apart in real life.

The Lego Challenge

What initially seemed like a straightforward simulation problem quickly revealed itself to be deceptively complex. Creating a realistic snapping and unsnapping system for Lego in a virtual environment required solving several fundamental architectural challenges that most 3D engines simply weren’t designed to handle.

Data Structure Mismatch: The first major hurdle was structural. Lego constructions don’t follow a hierarchical parent-child relationship - they’re essentially grids of interconnected pieces where each brick can connect to multiple neighbors simultaneously. This conflicts with how virtually all 3D engines (including Unity) organize objects in tree-based hierarchies.

Rotation and Pivot Points: From a 3D graphics perspective, parent-child relationships are essential for proper rotation around pivot points. We needed to maintain Unity’s tree structure for rendering while simultaneously managing Lego’s grid-based connection system - essentially maintaining two different data structures in parallel.

Dynamic Unsnapping: When a user removes a Lego piece, all bricks sitting on top of it must also detach. This chain reaction needed to be calculated dynamically and in real-time, determining which pieces were still supported and which should fall away.

Group Fragmentation: Perhaps the most complex scenario: removing a single piece could split one Lego construction into multiple separate groups. The system needed to dynamically identify and manage these newly separated structures, determining which pieces belonged together and which had become independent objects.

Physics Simulation: Beyond structural management, each Lego piece required accurate physics simulation for weight, gravity, and collision behavior. Bricks needed to feel solid when snapped together but respond naturally when falling or bouncing after being separated.

Software packages used:

  • Unity3D (5-UDE)
  • Blender
  • Maya
  • Photoshop

Credit:

  • Anvit Srivastav - 3D Modelling
  • Andrew Spies - Developer
  • Ravi Nankani - Developer
  • William Broderick - 3D Modelling & Level Design
  • Lesley Pandian - Developer\Team Manager