Piano Hero VR
By Elliot Fiske
For my final project, I explored the various ways peripheral hardware can be brought into VR. Specifically, I brought my MIDI electric keyboard into VR and created a simple "Guitar Hero" game where users play notes as they fly towards the corresponding key.
Everything was created in Unity, and the entire project is available at this Github repository.. For the scene, I used a free Piano model above a free wooden floor texture. The room is dark except for a spotlight illuminating the player, and the chatter of a crowd can be heard close by in the darkness. Here is a video of the application in action:
As you can see, if you miss notes, the "Tip" meter goes down, and eventually the audience starts booing you. If you perform the notes correctly, your score will go back up. A screen next to the piano shows the stream of the Vive's front-facing camera, which can be useful to orient yourself between the real world and VR.
I used the free Unity package MidiJack to receive MIDI input from the controller, and Newton VR to create a realistic physical VR world. Each of the piano keys use a hinge joint with a spring to model how the keys work in real life. Because of this, if you interact with the keys by smashing them with the Vive controller, it will match closely to how the keys react in real life.
In order to calibrate the VR piano's position to its real world position, I used a variety of techniques. First, to get the right angle, the user held the controllers parallel to the piano and pressed a trigger, as you can see below:
We need the VR camera to be facing perpendicular to both the piano, and the world up vector. Thus, in order to get the right angle, we set the camera's base front vector to AB crossed with the world up vector. Also, to ensure the whole world doesn't get tilted, we first project AB onto the y = 0 plane.
In order to fully calibrate the piano, you can also adjust the X, Y and Z positions of the virtual piano to get it to perfectly line up with the real piano. You do this using the trackpads on both of the controllers.
