Mixed Reality with Keyboard and Kinect

Finally! Tango Chen (aka Jingzhou Chen) is back with a new Kinect project!

MIDI Reality – Mixed Reality with Kinect and MIDI Keyboard


"Tech & music are my favorites. I’d love to do them together.

So I had this idea to try covering a song with a MIDI keyboard, and a Kinect v2 sensor for mixed-reality floating caption effects.

That was when the song (Life Is Worth Living by Justin Bieber) was still new. However, the recording process’s been so procrastinated.

Since it had taken for a very long time, I decided to not making a better cover, but this somewhat bad, weird and incomplete one. And, explaining how that was done more focusedly.

You can consider it as how most mixed reality / augmented reality effects are done behind nowadays. Literally overlaying parts of(or full) virtuality on a reality layer. I can not be 100% sure though, cause I haven’t been looking into any references for this project.

Furtherly, using regular AR marker technology to track position of MIDI keyboard, Kinect can capture around for a movable camera view(Requires a stored 3D background data also). Or just use HoloLens!"

Project Information URL: http://tangochen.com/blog/?p=2337

Contact Information:

from Channel 9 https://channel9.msdn.com/coding4fun/kinect/Mixed-Reality-with-Keyboard-and-Kinect

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s