Pedestrian Simulation

A pedestrian simulation for visionOS that simulates a dangerous traffic scenario

This project was developed as part of my bachelor thesis, which researched the possibility of building three-dimensional pedestrian simulations for the Apple Vision Pro. According to the World Health Organization's global status report on road safety 2023, pedestrians still account for 21% of the global road traffic fatalities, underlining the need for better tools to study and understand their behavior in dangerous traffic scenarios. This simulation is a proof of concept which aims to demonstrate that the Apple Vision Pro is capable of addressing this need, by allowing researchers to simulate realistic traffic scenarios in a controlled environment. The simulation places the user at a crosswalk with the objective to cross the street in order to reach a bus on the other side. As the user walks across the street, a high-speed car approaches from the left and stops right before the crosswalk. Since the car is hidden from view by multiple objects, users are only able to see the car at the last second, showcasing the Apple Vision Pro's potential for recreating realistic traffic scenarios that are too dangerous to stage in real life.

The Apple Vision Pro is particularly well suited for this kind of behavioral research. Unlike most headsets, it combines an exceptionally high display resolution with the ability to run demanding applications natively, without the need for an external computer. Through a complex 3D scene, ambient and spatial audio, realistic lighting, and dynamic interactions between the user and the environment, this proof of concept demonstrates that the Apple Vision Pro is a viable platform for pedestrian safety research, especially for simulations that require a strong sense of realism and immersion. By allowing multiple participants to experience the same critical traffic scenario, the Apple Vision Pro could further help researchers understand how people navigate in these dangerous situations and ultimately support the development of more advanced road safety systems.

Warning

This software is NOT a consumer product, a game, or a validated scientific tool. It is an academic proof of concept. You must carefully read this document and the SAFETY.md before attempting to download, compile, or run this software, or before facilitating its use.

Demo

Pedestrian-Simulation-Demo.mp4

Important

This demo runs in the visionOS Simulator. Subsection 4.2 and subsection 4.3 of the SAFETY.md do not apply here. Therefore, this demo is not representative of using the app on physical Apple Vision Pro hardware, where all safety requirements described in the SAFETY.md apply.

How this simulation was built

When I started this project, choosing the right toolchain was more difficult than I had anticipated. At the time of development, visionOS was still a relatively new platform, and the game engines that supported it either required an expensive license or did not provide the features required to build an immersive simulation, which led me to take a different approach entirely. Rather than relying on a game engine, I modelled the static scene in Blender and imported the final USDC file into Reality Composer Pro, Apple's dedicated tool for RealityKit content. Reality Composer Pro then helped me bring the scene to life by allowing me to add animations, a sky dome, and various audio sources, which together gave the environment the depth and atmosphere it required. This approach allowed me to develop a working proof of concept, demonstrating the Apple Vision Pro's potential as a research tool for studying pedestrian behavior in realistic traffic scenarios.

However, having the simulation work correctly on the Apple Vision Pro required navigating several challenges. To begin with, VR applications under visionOS are primarily designed for stationary use and enforce a 1.5-meter movement limit, making the full immersion mode unsuitable for a simulation that requires users to walk freely. Mixed immersion mode ended up being the appropriate choice because it removes this limit while still helping users remain aware of their physical surroundings by making nearby content semi-opaque if they move too close to a physical object. Likewise, the window management under visionOS required similar attention, because the platform does not permit the last open window to be closed, and opening a new one causes it to appear offset from its intended position. Therefore, I had to write a combined View called InteractiveWindow.swift instead of creating separate Views for the individual text windows. Lastly, having the high-speed car arrive from the blind spot at the correct time also required a creative approach, because the simulation had to react to the user's movements. In a true game engine, this is possible using colliders, but since Reality Composer Pro is not a game engine and provides no native way to detect collisions between the user and virtual objects, I came up with a different solution. I ended up writing a class called DevicePositionProvider.swift to retrieve the user's position within the scene's coordinate system and play the animation after the user has traveled a fixed distance from the starting point.

With the simulation functioning as intended, the remaining challenge was ensuring it provided the level of immersion that behavioral research requires. When studying how pedestrians behave in dangerous traffic scenarios, participants need to feel genuinely present in the environment for the data to be meaningful. This is achieved in part through audio, as the simulation uses ambient environmental noise to set the scene, and attaches spatial sound sources to the bus and the high-speed car so that both can be heard from their respective directions. However, one of the most difficult parts of creating realistic simulations for the Apple Vision Pro was lighting and shadows. While dynamic lights are supported, it is strongly advised by Apple to use light sources sparingly, as they can be computationally expensive. In order to get accurate lighting and shadows, Apple recommends using texture baking, which can be done in Blender. This process calculates the textures for all objects ahead of time and bakes the lighting and shadows into the textures themselves. In other words, lighting and shadows are part of the image texture that the Apple Vision Pro needs to load. While texture baking is computationally expensive during rendering, it is extremely efficient during runtime, as the Apple Vision Pro only needs to load a single texture and does not need to worry about computing lighting.

While texture baking generates very realistic results, the manual process can be extremely difficult for large and complex scenes. Depending on the number of objects and materials, it might take several days just for a small scene. That does not take into account the amount of time that is required to render the textures themselves. To speed up the tedious manual process, I wrote a Python script that allowed me to bake the complex scene of this project in just a few hours, instead of days or weeks. I used this script to publish a more advanced Blender add-on called OpenBake. This add-on aims to help anyone who develops their own simulation for visionOS and would like to make use of Blender's texture baking without having to go through the trouble of doing it manually.

How to install and run this app

Important

Due to GitHub's file size restrictions and Git LFS bandwidth limits, not all files required by this project are downloaded during a clone. Make sure to carefully follow the steps below to add the missing but required PedestrianSimulation.usdc asset to the project.

You must follow these steps to install and run this app on your device:

1. Carefully read this document and the SAFETY.md.
2. Install Xcode and clone this project.
3. Download and add the required PedestrianSimulation.usdc asset to the project by either:
   • navigating to the project's root directory and running ./Scripts/download_required_asset.sh or
   • manually downloading the asset from the latest release and placing it in the Packages/RealityKitContent/Sources/RealityKitContent/RealityKitContent.rkassets/ directory.
4. Build the app and run it on a simulated or physical device.

More information on how to install and run a visionOS app can be found here.