Head Tracking for Desktop VR Displays using the WiiRemote

Check out Johnny Lee's awesome "Head Tracking for Desktop VR Displays using the WiiRemote"! ▶ this video More great stuff: link

Specifically: ▶ this video Check out his entire youtube channel, it is full of interesting software/hardware hacks.

> ▶ this video I did this at the time. I downloaded his Wii app that shows the little red/white discs, and bought a battery powered Wii-bar thing that had the IR leds in it and strapped it to my head. Can confirm, the 3D effect is 100% spot on, even the smallest move of your head caused the display to adjust perspective and your brain is totally fooled into thinking the Wii display is a just a window into a 3D scene. tbh, it was the only cool thing I actually did with my Wii!

Wouldn't it be better to use head tracking to get the position of the head relative to the monitor, so the monitor behaves like a window? Like in Johnny Lee's classic Wii demo [1]. The way it currently works (rotating the view upon head rotation) doesn't really make sense because a monitor is not a head mounted display. 1: ▶ this video

You're probably recalling OG Johnny Lee video: ▶ this video That's cool, what were the benefits/drawbacks of that compared to other VR headsets would you say?

I tried to implement Johnny Lee's amazing idea ( ▶ this video ) using mediapipe face tracking. I could not move much far using simple webcams since it was getting difficult to determine the distance of the face from the camera when the face was turned. I had an Inel RealSense 415 depth tracking camera from a different project and it took care of the distance thing at least. But the jitter thing had me stumped for a long time and I put the project away. With your ideas, I get the strength to revisit it. Thanks!

... or the various projects by Johnny Lee [1], using the then new wiimote in completely novel ways [2][3]. 15 years later, I'm still surprised these proof-of-concepts didn't evolve into something more commercial. [1] link [2] ▶ this video [3] link

You need to use the Wii remote in reverse so you can make a fake window. ▶ this video (can't believe this was 15 years ago now..)

I absolutely agree. I'm very much a whiteboard hand-wavier guy. I explain with gestures as much as I do with the actual drawing (might be an Italian thing too). That said, this "pancake" problem reminded me of the old Wii head-tracking project.[1] There might be an interesting hybrid opportunity with that idea and 3d avatars. I feel like the head-tracking hack never really took off because it only worked as a solo experience. But since remote work is mostly us all individually sitting at a computer I could see it working better. Combine this idea with a large dedicated monitor/tv and now you have something that would literally just feel like a window/portal to the person you're talking to. [1] ▶ this video

Cool, but, it looks exactly like watching a display with a Wii remote on your head [0, 2007]. What’s so special? Is it actually 3D this time? [0] ▶ this video

"We offer an experience of 3D on a single 2D image using the parallax effect, i.e, the user is able move his real-time tracked face to visualize the depth effect." Since there are no examples I can't be sure if this is is what I think it is, but IF it is: I want this on huge monitor for any 3D game instead of clunky headgear VR or tiny smartphone AR. Months ago I also tested this with a small 3D visualization and very crude head tracking. The effect is damn awesome! To be able to move around in real space with the rendering adapting to it, makes it so immersive, even for my very crude tests. In my opinion the resulting 3D effect is MUCH better than viewing in stereo with one picture per eye. Here is an example from someone else from 2007: ▶ this video From 2012: link Obviously this works for only one person viewing it, but does that really matter? There are a LOT of use cases where only a single person uses a single monitor for viewing, especially in these times. In fact it is the standard.

What's interesting about this is that the tech has been proven dating back all the way to 2007 [1] and also re-implemented by enthusiasts many times over the years with better eye tracking via CV. What Sony has done here is refine it and package it in a way that's nice for end users. Personally, I'd like to see this implemented more cheaply in larger displays to act as virtual windows for indoor environments, like [2] from 2010, but with 3D video/rendering. [1] ▶ this video [2] link