Web based application to visualize Dani in AR playing piano

This time you can see an AR web based application, using ARKit and Three.js to scan a marker and render some 3D models. 

Let’s talk about the 3D models shown:

• Dani (a 3D avatar of myself): In order to create, animate and render this avatar I scanned myself with the “Structure Sensor” and converted it into a mesh with Itseez3D. Then, I rigged this mesh (added bones) to animate it and for this specific part, I developed an algorithm in colaboration with Merce Alvarez, which automatically rigs your 3D avatar (it must have a humanoid appearance). Finally, in order to get the 3D animation, I have used one from Mixamo, since creating my own one using Oculus Rift wouldn’t have had finger tracking.

(Note: Complex animations out of real movements can be created with “Perception Neuron” or combining HTC Vive with Leap motion and two extra sensors for your feet).

• 3D piano and bench to sit. The first one I downloaded with a free art license and about the bench, I have modeled it using Blender. Then, I  have added an armature to both so I could play them along with my avatar in the web browser.

Once the models were ready, I prepared the AR marker in a way such that you can scan a QR code inside the marker, to get the link of the application and, from there, scan the AR marker. For that, I have taken a screenshot from the animated models, edited it with Photoshop and turned it into borders only, so it doesn’t overload the marker.

The resultant marker can be seen below:

• 

You can try it yourself scanning the QR code, and then, the whole marker after opening the website. If you can’t manage to scan the QR code, find the AR web-application here (open it with your phone or laptop): 

Finally, below a video of the experience (music was added afterwards, played by me. It could be added to the AR application though).