HN Theater @HNTheaterMonth

⬐ bckr

I remember this. I was a teenager and this inspired me greatly to learn more about tech.

⬐ wingmanjd

I still have my IR light-pen somewhere that I bought for his Wii Whiteboard app[1]. This was just a few years after the Minority Report movie, and multi-touch was still something that felt super sci-fi to me at the time. Seeing it come alive with just a wiimote and an IR pen was neat.

1. http://johnnylee.net/projects/wii/

I absolutely agree. I'm very much a whiteboard hand-wavier guy. I explain with gestures as much as I do with the actual drawing (might be an Italian thing too).

That said, this "pancake" problem reminded me of the old Wii head-tracking project.[1] There might be an interesting hybrid opportunity with that idea and 3d avatars. I feel like the head-tracking hack never really took off because it only worked as a solo experience. But since remote work is mostly us all individually sitting at a computer I could see it working better.

Combine this idea with a large dedicated monitor/tv and now you have something that would literally just feel like a window/portal to the person you're talking to.

[1] https://www.youtube.com/watch?v=Jd3-eiid-Uw

⬐ usrusr

I'd expect conventional live footage of whiteboard hand-waving to wildly outdo VR for the entire foreseeable future. If you need two guys hand-waving in different rooms, focus the extra tech on the whiteboards, to somehow merge their content.

⬐ marcind

plug: this is along the lines of our thinking, at ShareTheBoard. rather than kill the whiteboard with VR, we use vision to give it new tricks.

step 1 is real-time content digitization and obstacle avoidance (available now). step n involves synchronizing cameras and projectors for a deviceless AR experience - true "remote whiteboarding." we've completed this in laboratory conditions, hope to release it next year.

⬐ vyrotek

Is it possible to use this with Teams or Zoom or does everyone need a new app? This seems like an ideal plug-in or app to use with those since it's essentially just a "filter" app from the presenter's side. Using your app as the video source seems like it'd be straightforward to do.

⬐ marcind

Indeed, most of our users simply open our app, then use Teams/Zoom/Meet to share that tab using the built-in screen-share. That gives you content legibility but none of the other benefits: by hitting our meeting link (in browser: no sign-up/download needed), all viewers can also save board contents at any time and even add digital content of their own.

Can't do that second part "through" videoconferencing apps - not without a deeper integration (read: participation with the companies in question). It's on our list of possible developments but will require dancing partners on the other side.

This looks great for if you want multiple people viewing something, but I wonder why these [1] single person holograms never took off.

[1] https://www.youtube.com/watch?v=Jd3-eiid-Uw

⬐ vernie

Sony recently launched a product based on this concept: https://electronics.sony.com/spatial-reality-display/p/elfsr...

⬐ nomel

Those single person "holograms" aren't 3d. They offer a perspective shift that tracks a single point on your head, and look great in video, or with one eye closed, but are 2d when viewed with the eyes. For 3d, you need to give each eye a different perspective.

These holograms are 3d, since each eye is presented with an image that matches its own perspective.

⬐ E-Reverance

Good point. Although if you combine this with 3D TVs, then you basically get a single person hologram, right?

⬐ nomel

Yeah, that's how this system works, using eye tracking instead of a goofy game controller strapped to your head ;) https://www.cnet.com/tech/computing/sonys-5000-eye-tracking-...

⬐ TaylorAlexander

Sounds like it, but “it’s 3D if you wear these glasses and you’re the only person viewing the screen” is a pretty limiting use case.

⬐ rtkwe

Only for one person though which isn't a great thing for TVs which generally have more than one person watching. Also the tracking is a bit of an issue, you'd have to wear a little headset to accurately track your head when they were released. Now you could maybe get away with using a camera for tracking but now you've got the privacy worries of that happening.

I have very little knowledge of this, but as a product would it not be much cheaper to have something like a wii bar fixed to the top and bottom of a screen, with a small camera at the extremities of each bar (one for each of the four corners)?

You could then use the camera perspectives to create a 3d image of the person you're conversing with and map the colour data correctly to that 3d image (Photogrammetry)

You could also likely use the information from the four cameras to map the orientation of the 3d image of the person you're speaking with to give you that sense of depth as you shift your position. https://www.youtube.com/watch?v=Jd3-eiid-Uw

If you had a speaker and mic in each corner you could also capture / emit subtle differences in audio to further enhance it.

Cool, but, it looks exactly like watching a display with a Wii remote on your head [0, 2007]. What’s so special? Is it actually 3D this time?

[0] https://www.youtube.com/watch?v=Jd3-eiid-Uw

⬐ Jemm

TrackIR and variants are quite popular for simulator type games like flight simulator or truck simulator. They let you look around in the game just by moving your head.

⬐ dividuum

Hm. I haven't really looked too deep into TrackIR, but I always thought it tracked your head movement to simulate head rotation within the game. So you could look around more naturally. The linked Wii "tracking" demo used the tracking to find your position relative to the display to change the rendered perspective accordingly. Does TrackIR also allow this?

⬐ winthrowe

As a TrackIR user, it feels to me like it should be possible to do something like this with the 6dof headtracking data from TrackIR, but basically nobody uses a 1-1 mapping that would produce a camera trick like this.

⬐ mrguyorama

TrackIR DOES do this. In flight simulator, with full 6dof, you can move your virtual head around and look under and around things

⬐ winthrowe

In my TrackIR config, I move my head left an inch, the projected head moves 6 inches. This is more useful than a even mapping in practice, but solidly breaks the 'looking through a window' illusion.

⬐ detaro

> What’s so special?

That it's not based on head tracking, and in a product you'll be able to reasonably buy.

⬐ mnw21cam

I'm just going to link https://www.cl.cam.ac.uk/research/rainbow/research/autostere... here - an autostereo display that was working way earlier than that. "The first displays were built in the late 1980s and early 1990s". They worked by projecting different images out to different horizontal areas in front of them, so your right and left eyes would see a different image, you could move your head and see around objects, and multiple people could look at it and see an appropriate view for their position.

Not sure if they ever ran Doom on it though.

⬐ jcelerier

with an holographic display two persons looking from different angles would see the image corresponding to their angle

⬐ mdip

I think that sums it up pretty perfectly. It's the single reason I watched the video and my first thought was "there's no way that works that well"

From the comments, here, it sounds like it might. There's some interesting applications, here. Thinking back to last night, it would have been really nice if when I was looking at the split-screen in my driving game with my kids, my own corner had that kind of depth to it. Hell, driving/flying games, in general, suffer from realism issues due to not being able to faithfully produce the effect of looking out a window.

Doing this on a larger screen (and throw in a transparent LCD after you work out how to bend all of this into that, assuming it can be done) and you could have a light-switch that changes the scenery from "outside" to "outside somewhere else"[0].

[0] I remember seeing that in Back to the Future II as a kid and thinking it'd be really nice to have on a cold, gloomy, February day in Michigan.

⬐ runawaybottle

Well, if it’s as simple as head tracking with a IR emitter on your headphones or the like, then I’m a little surprised first person shooters don’t enable peeking around corners with head motion.

⬐ xwolfi

It doesn't seen to be, or rotating the thing with a fixed camera wouldn't produce the effect, I guess ?

https://lookingglassfactory.com/tech

⬐ mattowen_uk

> https://www.youtube.com/watch?v=Jd3-eiid-Uw

I did this at the time. I downloaded his Wii app that shows the little red/white discs, and bought a battery powered Wii-bar thing that had the IR leds in it and strapped it to my head.

Can confirm, the 3D effect is 100% spot on, even the smallest move of your head caused the display to adjust perspective and your brain is totally fooled into thinking the Wii display is a just a window into a 3D scene.

tbh, it was the only cool thing I actually did with my Wii!

⬐ mdip

Thanks. I just created a nice storage box for my Wii, got everything organized and put it on the bottom of a large pile of things to sell.

This sounds way too interesting, so I'll be doing some digging, apparently. Any idea if this would work effectively on a large projector screen?

⬐ mattowen_uk

> Any idea if this would work effectively on a large projector screen?

Funny you should ask that! My set up IS a large projector screen (110"), and yes it does work. In fact the size enhances the experience. :D

⬐ mdip

Ha! I'm planning to do the same on the 110" I've got in the basement.

⬐ SirHound

Surely the depth effect only worked with one eye closed?

⬐ wffurr

Nope, works with both eyes.

Your brain takes a lot more cues for depth than stereoscopic vision. The way the perspective changes as you move your head around is sufficient even for a flat display. It’s uncanny.

Only works for one person though. If you stand at an off angle to the sensors, there’s no effect at all.

⬐ RHSeeger

Its amazing how the brain works...

I am very cross-eyed at the moment (surgery resulted in both my eyeballs moving) and require a little plastic "stickon" that goes on the left lens of my glasses; which has prisms to redirect the light so that both eyes are looking in the same direction (prisms are fairly normal; it's the _amount_ of correction I need that is abnormal).

The correction on the left lens is so bad that looking through it is completely blurry (with lots of rainbows if I'm looking towards a light). However, when looking through my glasses with both eyes, everything appears normal. I can see in stereo (which I cannot do w/o the correction) and everything is in focus. There is still _some_ rainbow effect.

So, my brain is taking the "clear image" from my right eye and overlaying it with the distance information that the left eye adds to that... but ignoring the blurry image from the left eye completely.

Honestly, I find it mind boggling.

⬐ cbm-vic-20

And that video is over 13 years old! Too bad this idea didn't catch on more widely.

⬐ mrguyorama

TrackIR is a consumer product using the exact same tech for headtracking in video games. I think it might actually be older than the Wii

⬐ Wowfunhappy

It doesn’t seem to exist anymore, but I had a variant (literally same graphics) on my iPhone a decade ago. No fancy headgear and the effect was just as good—but instead of moving your head, you held still and tilted your iPhone.

Apple later added the effect to the homescreen in iOS 7, but much more subtle.

Games made to take advantage of hardware innovations that were then built upon. Nintendo has had a direct hand in the implementation of new (or newly low-cost) technologies that have then gone on to serve as the basis for more widespread innovations. Ex. They placed a cheap infrared camera in their Wiimote to enable controller tracking (which means that that tech got into the hands of millions of users). An engineering student reversed this, using that camera to track LEDs placed on a pair of glasses to achieve cheap headtracking.(1) This was then used as the basis for the tech in Microsoft's Kinect(2), which served as the basis for the tracking technologies we see in modern XR setups. That's just one example of many.

1: https://youtu.be/Jd3-eiid-Uw

2: https://www.wired.com/2011/05/johnny-lee-kinect-hacking/

"We offer an experience of 3D on a single 2D image using the parallax effect, i.e, the user is able move his real-time tracked face to visualize the depth effect."

Since there are no examples I can't be sure if this is is what I think it is, but IF it is:

I want this on huge monitor for any 3D game instead of clunky headgear VR or tiny smartphone AR.

Months ago I also tested this with a small 3D visualization and very crude head tracking.

The effect is damn awesome! To be able to move around in real space with the rendering adapting to it, makes it so immersive, even for my very crude tests.

In my opinion the resulting 3D effect is MUCH better than viewing in stereo with one picture per eye.

Here is an example from someone else from 2007:

https://www.youtube.com/watch?v=Jd3-eiid-Uw

From 2012:

https://www.youtube.com/watch?v=h9kPI7_vhAU

Obviously this works for only one person viewing it, but does that really matter? There are a LOT of use cases where only a single person uses a single monitor for viewing, especially in these times. In fact it is the standard.

⬐ zenir

I actually hacked something together like that and it surprisingly looks much better in a video than in real life. I guess the main reason being that we have two eyes and the brain still realizes it is flat (which is weird because it doesn't look flat in a video).

⬐ vanderZwan

It works well in the video because in that context we are trying to estimate depth relatively to the camera.

In real life, we are trying to estimate depth relative to us. In order to fool our brains we need both a very low latency and a high frame-rate. That was one of the major hurdles to solve for VR as well, leading to John Carmack's famous complaint that he can ping across the Atlantic and back faster than he can send a pixel from his desktop to his screen[0][1].

Anyway, back to the video: basically, when comparing 3D movement relative to the camera our brains seem to be more "forgiving".

[0] https://twitter.com/id_aa_carmack/status/193480622533120001

[1] https://danluu.com/latency-mitigation/

⬐ karmakaze

Thanks for posting these similar demo links. I have a problem with visual programs/libraries that reference no examples: demo or GTFO. I realize that the repo may have been posted here by anyone but it's a good note to reference a demo if one is/can-be-made available.

⬐ fish44

looks like it uses this https://shihmengli.github.io/3D-Photo-Inpainting/#example which makes this super cool...

⬐ sjs382

I first saw that in person on a pinball machine like this: https://www.youtube.com/watch?v=64e7TQ5uj8g

The effect is fantastic.

⬐ RobertoG

I wonder if this could be use for better videoconferencing.

⬐ matsemann

I remember being awestruck by that 2007 Wii video, and spent a lot of time playing around with the remote after that. That, and the Xbox Kinect, were really cool tools. Why hasn't those concepts seen more widespread use outside gaming?

As for 3D, I also remember seeing a TV at Toshiba HQ in 2013 or so that had 3D without glasses, and even worked for multiple people. No idea how.

⬐ numpad0

Sony has a head tracking OLED sub-display for VR/3D content creation in office setups, Looking Glass was doing crowd funding recently for an apparently very well done lenticular digital photo frame, those two comes off top of my head.

Lenticular 3D limits resolution to a fraction divided by double of the number of viewpoints in an axis, e.g. a 4K by 2K panel with 10 viewpoint along its width means effective resolution is 400 by 2K. Okay for demos but not practical atm.

⬐ vanderZwan

> Why hasn't those concepts seen more widespread use outside gaming?

I wrote my master thesis on designing gesture interfaces (of the Xbox Kinect type) back in 2012, here's my two cents:

First of all, gesture detection still isn't reliable enough for serious input. Missing a beat one in a hundred times is acceptable in gaming settings (and even then only in more "casual" environments like party games), but for serious input the input device needs to be practically 100% reliable. A keyboard press is. A mouse click is. Detecting whether your hand is an open palm or a fist? Not so much. Hence the peripherals we typically see for VR games, which help of course. At the same time they also somewhat defeat the purpose.

A second major issue is a lack of haptic feedback. There is no such thing as touch-typing in the air.

Why this is such a big problem needs a bit of explaining: a practical way to think of our ability to manipulate our environment (literally, the manos in manipulate referring to our hands) is to think of them as a pair of kinematic chains[0][1]. Essentially this is a chain of ever-more finegrained "motors", going from coarse-grained to fine-grained precision: our shoulders, our elbows, our wrists, and finally the digits of our hands. The ingenuity of this chain is that it allows for extremely fine precision (the sub-millimeter precision of our fingertips) in large spatial volume (the reach of our arms), and it does so by having each "link" in the chain perform a bit of "error-correction" for the lower resolution of the previous link.

What does this have to do with gesture interfaces? Well, in order for that kinematic chain to work, it needs a precise feedback system to perform said error-correction. We basically have three senses for this: our visual system (that is, seeing where we are putting our hands), our haptic sense (feeling which button we're pressing with our finger-tips) and our "spatial sense". The problem with the latter sense is that is relative: I sense the sub-millimeter location of my fingers relative to my wrist. I sense the millimeter-precision location of my wrist relative to my elbow. I sense the centimeter-precise location of my elbow relative to my shoulder. So if I'm waving my hands in the air without looking, the effective "precision" they have is about as crude as the crudest link in the chain: my shoulder. Of course this spatial sense can be improved with training, but you know what we typically call people who are really good at that? Professional-level dancers. The ceiling of mastering this skill is pretty high, and there's a reason it's basically a profession all by itself (plus a ton of other things obviously, don't want to sell dancers short here).

Gesture input also will never be as easy on the motor skills as typing: not only does a keyboard provide the haptic feedback from the keys, the precision of my fingers is relative to the wrists that are resting on the desk, not to my shoulders.

Games somewhat get around this by representing a visual avatar to give us feedback, but it's not perfect. On top of that, this feedback is limited by the resolution of the gesture detection, which is ludicrously low compared to the potential precision of our limbs. And if that wasn't enough, it also needs a really low latency to fool our brains and really "feel" like an extension to our senses.

So basically, the fidelity requirements are just brutally high.

And finally, there is only a limited set of use-cases. There are basically just two big ones: "touchless" interfaces (very niche) and pointing and manipulating in 3D space (less niche, with a clear advantage over keyboard or even mouse input, but again having brutally high fidelity requirements). Because of that, as cool as gesture interfaces are, the industry-wide drive to solve all the aforementioned issues just isn't quite as high as we'd like it to be.

[0] http://cogprints.org/625/1/jmb_87.html

[1] https://en.wikipedia.org/wiki/Kinematic_chain

⬐ sbarre

What are your thoughts on peripherals that provide haptic feedback to help solve the feedback loop?

I am thinking gloves or other wearables that could provide force-feedback to help us correlate our physical actions to the virtual interface..

Are we making progress in those areas at all?

⬐ vanderZwan

> What are your thoughts on peripherals that provide haptic feedback to help solve the feedback loop?

Heh, sorry to disappoint but I've not done IxD design work/research of that kind since graduating, and not actively kept up with developments. Basically my well-informed thoughts are limited to "one of these days I'll save up the time and money budget to get a VR system, a gaming desktop powerful enough to run it, and the peripherals, to try those out and be overly critical of them, and I am fully aware that I'm lying to myself right now."

I'll indulge myself with some big-picture speculation though - but I want to emphasize that it's just that: speculation!

I'm sure that there is progress being made, but I suspect the major breakthroughs won't be made through VR, or at least not the gaming side of it. The reason for that is that game design is built on top of existing hardware, which in turn drives the need for more specialized hardware, so ultimately that will result in more specialized, niche tools.

Let me expand on that: we're talking about making "progress", but progress by which metric and for which purpose? What is "progress" in a VR game context? (To tie this into the peripherals you suggested: those are one possible solution, but to which problem exactly, and is that really a problem we're struggling with?)

Let's take graphics in games as historic point of comparison. For decades, marketing has tended to focus on "realistic", "next-gen" graphics. But given that a game with abstract pixelated graphics can still be extremely engaging, and that a game with extremely realistic graphics can still be boring, we might be over-estimating the value of realism in games there. To make an analogy-within-an-analogy: it took Ancient Greeks just one century to go from copying Egyptian sculpture techniques to near-perfect human anatomy[0]. Where did they go from there? Exaggerated beauty ideals! They mastered realism and then decided to ignore the parts of reality that didn't interest them.

How does that insight translate to input devices? I can't say for sure of course, but given that mouse-clicks have never felt like an obstruction to immersion, I suspect that the most fun gesture-based games might lean a bit less on realism than we expect. For gaming input we want to feel in control of our gaming avatar. So low-latency still is likely to be important, as is "perfect" input detection is important.

However, and this is my main point: we never stated that "perfect" input detection has to be realistic input detection! A lot of game input schemes are more about giving us the feeling of control and mastery of skill than "realism", whatever that means in this context.

So I wouldn't be surprised if gaming will turn out to be a dead end for generic interface innovations as a result, since the interfaces of games will only become more specialized to the purpose of making them engaging. No disrespect to game UX - there is a lot that general interface designers can learn from studying game design, since at its core it is about building aesthetic interactions. But it's not quite focused on solving the same issues.

I would place my long-term bets on immersive computing environments like Dynamicland[1] coming up with truly novel, more generic interfaces that break out of this problem, due to it having more use-cases for generic 3D input. But we'll have to wait and see (or join the people at Dynamicland and try to actively make progress in that department).

[0] https://en.wikipedia.org/wiki/Ancient_Greek_sculpture

[1] https://dynamicland.org/

⬐ numpad0

There were some people attaching some things other than pistol grips to Novint Falcon for use in VR in DK2 era but never heard the company went back to resume production.

⬐ unholiness

Back in 2012 I did some building on ZSpace[0] and the that setup still feels like the sweet spot: Physical keyboard, physical pen (with virtual extension), physical glasses to detect head position for parallax, 3D environment to play and create in.

The _visual_ feedback from moving your head and rotating objects with the pen were extremely low-latency. Gesture detection is still nowhere near that level of fidelity but with peripherals, perhaps it's not necessary.

[0] https://zspace.com/

⬐ vanderZwan

Cool, that sounds a lot like Bill Buxton's experiments with two-handed input, but in VR!

What's interesting about this is that the tech has been proven dating back all the way to 2007 [1] and also re-implemented by enthusiasts many times over the years with better eye tracking via CV. What Sony has done here is refine it and package it in a way that's nice for end users.

Personally, I'd like to see this implemented more cheaply in larger displays to act as virtual windows for indoor environments, like [2] from 2010, but with 3D video/rendering.

[1] https://www.youtube.com/watch?v=Jd3-eiid-Uw

[2] https://hackaday.com/2010/04/16/virtual-windows-that-track-a...

⬐ ginko

From what I understand the display is still stereoscopic so you will see a separate image for each of your eyes. Johnny Lee's demo was just a regular image with view-tracked parallax. It looks really impressive in a 2D video demo, but you'd immediately notice there's no depth if you saw it in person.

⬐ moron4hire

Nintendo's 3DS did this.

⬐ mimixco

I think you have it exactly backwards. Lenticular lenses only work when you see them in person, and that's where they do have depth. Kids toy's (like in Cracker Jack boxes) are printed with the same technique and they have depth in person. Because the capturing camera and your own viewing screen show the same image to both eyes, the depth effect is lost in a "2D video demo."

⬐ soulofmischief

I was hoping the first link would be Johnny Lee... He was a true pioneer and hacker in this space.

⬐ deltron3030

I didn't click because I was expecting it to be the Johnny Lee Kinect hack, haha.

This reminds me of https://www.youtube.com/watch?v=Jd3-eiid-Uw (2007) Johnny Lee's demonstration using head tracking with infrared led's and a wii remote to implement the same thing. Obviously that limits the effect to the individual's head being tracked, but its amazing its been 12 years since that demo and we're still trying to commercialize the same affect.

⬐ ksaj

That was a great video, and highly hackerific. "That would be a bit goofy" is an expression you don't hear very often these days.

If you're alone, you could use this "one simple trick" (from 2007) to up the realism by quite a bit:

https://www.youtube.com/watch?v=Jd3-eiid-Uw

It would look goofy for a copilot, but it would make a world of difference for the pilot.

⬐ mrguyorama

FSX has support for trackir[0] built right in. For $150 you get head tracking. It's actually quite fun and was super useful for my budding simulator hobby

[0]:https://www.naturalpoint.com/trackir/

⬐ PaulRobinson

I'm wondering if you could now achieve that same effect with an iPhone or iPad and the FaceID sensors... no good for full-size flight sims but I'm wondering if this tech is now going to get its day some time soon.

This[1] was the first person I saw do this effect with a hacked Wiimote and some IR LEDs on his head. It's mentioned in the article. This definitely isn't new. However the good face tracking does make it easier. Here's a face tracking JS lib[2] that'll do it for you if you'd like to make it happen on the web. I've been playing with this (not free) library[3] to do face tracking to achieve this effect for AR and experimental interface purposes for like three months. It's got multiplatform support that is pretty good.

I worked this into an iOS display demo in ~2009 using their basic face tracking that made for like zombie face apps and shit. The RealSense is a MUCH more solid experience, but this is not new at all.

[1] https://www.youtube.com/watch?v=Jd3-eiid-Uw&t=164s

[2] https://tastenkunst.github.io/brfv4_javascript_examples/

[3] http://www.visagetechnologies.com/

Yo mark willson of co.design. Were you around 10 years ago?

Let's tame the article title a bit huh?

https://www.youtube.com/watch?v=Jd3-eiid-Uw

⬐ giarc

He talked about Johnny in the article.

>Norrby is far from the first person to play with trompe-l’oeil (also called “parallax”) in the modern age. Johnny Lee, now at Google working on augmented reality, first made his mark by turning a stock television and a Nintendo Wiimote into a wild 3D game.

⬐ jordache

oh ok.. so he should know it's not 'wild'.

annoying bait title.

This reminds me of this demo from 2007 https://www.youtube.com/watch?v=Jd3-eiid-Uw

And also didn't the Amazon Fire Phone have head tracking?

Also found this, done on an iPhone in 2011: https://www.youtube.com/watch?v=7saRtCUgbNI

Or this table using a Kinect in 2012: https://www.youtube.com/watch?v=2CbiOikirrg

This reminds of the head tracking demo using a Wii remote made by Johnny Lee years ago. https://www.youtube.com/watch?v=Jd3-eiid-Uw

⬐ mncharity

You can play with this in browser.[1][2][3] And there's OpenCV.js.[4] But I more often simply put a small square of yellow gaffer's tape between my eyebrows, and do color tracking.[5] Accurate, reliable, and cpu/battery cheap.

And for expert UIs, one doesn't have to emulate reality. So you might exaggerate parallax, be non-linear or discontinuous, etc. There might be a much larger vocabulary available, than merely "look through the small portal" and "make secondary content conditional on viewing angle". Maybe.

Why might you want this? Consider your primary "desktop" environment shifting into HMDs. But you still need to mix in screen time, to give your eyes a break. It would hypothetically be nice to have similar UI idioms on screen. Especially as your tooling becomes less and less like current 2D ones.

[1] old similar demo (broken on FF) http://auduno.github.com/headtrackr/examples/targets.html video: https://vimeo.com/44049736 [2] current tracking-only demo https://trackingjs.com/examples/face_camera.html [3] eye/gaze tracking (demos at bottom; but for me, they require good lighting, and beard removal) https://webgazer.cs.brown.edu/ using https://www.auduno.com/clmtrackr/examples/clm_video.html [4] opencv.js https://docs.opencv.org/master/d5/d10/tutorial_js_root.html [5] color tracking https://trackingjs.com/examples/color_camera.html

⬐ ladberg

This is mentioned in the article too...

⬐ Lewton

I'm torn, there's literally a link to Johnny Lee's work in the article. So your comment doesn't add anything.

But a lot of people read only the comments (or the comments before the article) so your comment does add value

Is it a good HN comment or not? :D

⬐ ChrisRR

Oops! I admit I skimmed the text and looked at the pretty pictures.

⬐ hungerstrike

Your own comment is so self-important that I stopped thinking about the article at all and moved onto thinking about how much I despise people.

So your comment adds less than nothing IMO.

⬐ Lewton

Agreed that my comment has little to no value

But I was genuinely curious how other people felt :)

⬐ bringtheaction

It’s an acceptable kind of HN comment. A link in the article is generally easy to miss and probably others missed it too.

Admittedly the article did talk enough about the other thing that it’d be pretty hard to miss if you read the whole thing, but I think we can forgive it still. Perhaps the parent to your comment opened the page and saw the video but didn’t bother to read the article. I think that’s fine too.

I always read comments first and in fact quite often I read the comments only. As long as we don’t derail the discussion completely by starting a top-level comment about how we perceive something based on the title alone I think it’s fine. That includes it being fine to mention things that you are reminded of by the title as long as you phrase it correctly. In my opinion of course.

⬐ mft_

LOL, remember getting that working at home, back in the day - gyrating in front of my monitor with (from memory) the Wii's IR transmitter bar held to my forehead.

It was a lovely little taste of what might have been.. although we're still waiting!

⬐ kakarot

I was more fascinated with his projected touchscreen using a pressure-activated LED at the end of a pen. He's quite the visionary.

Check out Johnny Lee's awesome "Head Tracking for Desktop VR Displays using the WiiRemote"!

https://www.youtube.com/watch?v=Jd3-eiid-Uw

More great stuff: http://www.johnnylee.net

Nice job! If you have a camera that tracks the viewer, you can also make the image it displays 3D as seen in this video from 2007: https://youtu.be/Jd3-eiid-Uw?t=2m46s (back then it used a Wiimote to track the head position).

⬐ rhaps0dy

Do you think a camera would be precise enough? This looks pretty awesome.

⬐ gmac

I had a quick go at this a few months back using Apple's CIDetector[1], and even at 'high' accuracy that didn't seem precise enough — it was distractingly jiggly.

[1] [CIDetector detectorOfType:CIDetectorTypeFace context:nil options:@{CIDetectorAccuracy: CIDetectorAccuracyHigh, CIDetectorTracking: @YES}]

⬐ Jack000

up close the lack of parallax would give it away, but I think it would work ok if the viewer is sufficiently far away.

Even more distracting than the shadows would be the fixed perspective. Unless you include some form of head tracking, the immersion will never feel real. Unfortunately, it'll limit one person per room.

Classic example of head tracking: http://youtu.be/Jd3-eiid-Uw

⬐ kevingadd

With 6 kinects in a reasonable configuration, I imagine you could achieve some decent head tracking. Probably not good enough to be convincing, though.

⬐ baddox

Using the same idea as 3D projection, you could support multiple people per room, but with reduced frame rate and the requirement that they wear glasses.

All the head moving made me think back to Jonny Lee's wiimote headtracking 3d display.

https://www.youtube.com/watch?v=Jd3-eiid-Uw

Perhaps Amazon have an eye tracking system which can do the same with out the need for the user to be wearing infra red leds?

⬐ unreal37

I've never seen that video before, so thanks for posting it.

Yes, the head movement in the Amazon video makes it look like head tracking/perspective changes built in to the device. But what for?

⬐ clarky07

Yeah I was coming here to say the same thing. Seems like it is definitely doing some sort of motion tracking.

⬐ tseabrooks

I'm actually thinking this is either "Kinect" on a tablet and/or eye tracking. The application specifically talks about additional sensors. being used to build apps. I'm not sure head tracking is a strong use case for additional apps. But something with the depth of MS's Kinect or eye tracking could be really interesting.

"- Johnny Lee and the ATAP-Project Tango Team"

Johnny Lee was the guy with the awesome Wii Controller demos back in 2007 (can't believe it's been that long).

http://www.youtube.com/watch?v=Jd3-eiid-Uw

edit: here's the full set of demos: http://johnnylee.net/projects/wii/ (also, I'm assuming it's the same guy, but his site says he's at Google now)

⬐ ChuckMcM

I was Google when they were trying to hire him, he turned them down to go work at Microsoft. Its interesting to see him at Google these days. I was really inspired by his projector/Wii hacks. I even built a couple with some of those pico projector units that Woot gets now and then for $100. My interest was building a simpler structured scanner for doing 3D printing.

⬐ dba7dba

Watching the demo in the 2nd half of the I thought, cool I can watch video in 3D without wearing goggles. And than I heard the head tracking really works only when 1 user is using it. Well dang it, I will sacrifice spending time with wife if I can watch a movie in 3D without wearing goggles...

I see that his focus moved from living room TV/game console to a mobile device...

⬐ coldsmoke

I tried out his demo back then and unfortunately it isn't as impressive in real life as it is on video. The problem is that at the distance you usually have your television the most important depth cue is the stereo separation of images, i.e. what you get when wearing 3D-glasses.

So while the objects move around as if you would move in relation to them - your brain still tells you that the image is painted on the surface of the television.

⬐ magicalist

Yeah, I was really disappointed at the time that Nintendo didn't move on it, as it obviously worked really well out of the box, with no hardware modifications needed.

It could have made an awesome additional experience to some games (even if it was optional), and they could have done it with no accessory required.

⬐ higherpurpose

I think he was tech lead for Kinect at Microsoft for a while, before Google snatched him.

⬐ cdelsolar

Awesome, I thought I recognized him! I was talking about him recently to a friend. Those demos were inspiring and very cool.

⬐ balls187

Pretty awesome to see what he's been up too.

⬐ yelnatz

I am so happy to see a familiar face when I played the video.

Back in 07/08 when he demoed his hacks in TED it really blew my mind. [1]

I was really hoping someone would pick him up and let him loose on some projects.

I guess Google did just that.

[1] http://www.youtube.com/watch?v=0H1zrLZwPjQ

⬐ None

None

⬐ bradyd

He worked on the Kinect project at Microsoft as well. In fact he put up the prize money for the Open Kinect contest run by Adafruit to develop drivers for the Kinect.

http://procrastineering.blogspot.com/2011/02/windows-drivers...

Actually, stereoscopic vision is overrated.

There are many hits our brain uses to make the 3D model of the world we have on our minds. Stereoscopic vision is one of them, but really just kick in for objects nearer than you arm cam hold. For distant objects, our brain uses other hints - that's why people aiming at far objects close one of their eyes.

One of the strongest is the parallax effect, specially for medium distance and relative distance between objects. If you doubt it, look at this video: http://www.youtube.com/watch?v=Jd3-eiid-Uw

Relative size and color fading works well for big distances - this is the prominent effect in parachuting videos.

These are only two mecanisms, and immersion is not only about 3D sensation. Full periphery vision is of great value for this (IMAX, anyone?). Sound hints, brigthness, contrast, and a lot of other hints also help our brain making the 3D model.

⬐ jacobolus

As you say, for far objects, stereoscopic vision doesn’t do much; however, that can be changed..

A really fun project I recommend anyone try is to make a sort of periscope for one eye [make one out of e.g a milk carton and a couple of rectangular mirrors, or some similar tube].

Leave one eye looking out normally, and make the other eye look through the periscope, aimed sideways. The result of this is to increase the effective distance between the two eyes from a couple inches to a foot or more (depending on your periscope).

Now try looking at distant objects like a cityscape or some mountains. The stereo effect will be increased dramatically, and the whole scene will feel very dimensional and close.

⬐ kirubakaran

So binoculars?

⬐ aylons

No doubt you can induce the same effect in far objects by exaggerating the difference between what each eyes can see. If you didn't have the size to compare, you would actually think they are nearer.

However, this is not very realistic - this is why sometimes the 3D effect in a movie vanishes or looks gimmicky. Until today, the most convincing 3D scene I ever saw were the recording in Avatar, which were recorded using a 3D camera very near to the face of the actor.

Johnny Lee's work with the WiiMote uses the same technique, and his demo is much more visually impressive. :-)

http://www.youtube.com/watch?v=Jd3-eiid-Uw

⬐ radicade

Author here. Very flattered for the comment about the WiiMote demo, as it is very much an inspiration and indeed an impressive and groundbreaking demo.

The key difference here is that in the WiiMote demo, the head-tracked location is used to shift the camera perspective so that 3d objects appear to have additional "depth"; whereas in anamorphic projections the effects of perspective are deliberately undone so as to flatten the image and appear to have no depth at all.

In the end they both use the same technique, but end up with different a outcome/effect. :-)

So damn clever... releasing an SDK for the Kinect[1] was one of the smartest things Microsoft could have ever done and I think has gone a long way to contribute to it being the fastest selling piece of hardware in history[2] (did anyone know that? I sure didn't, and barely believed it reading it).

This reminds me a lot of what Johnny Lee did with the Wii soon after it was released with head-tracking 3D[3]

Part of me wants to stop using a computer and take up beet-farming because the cool factor seems so much less than what these guys are doing.

Anyone on HN actively toying with a Kinect and want to share some video?

[1] http://research.microsoft.com/en-us/um/redmond/projects/kine...

[2] http://www.vgchartz.com/article/83375/kinect-is-the-fastest-...

[3] http://www.youtube.com/watch?v=Jd3-eiid-Uw

⬐ nitrogen

What I find cool is that this sort of application (3D video recording) is actually possible with libfreenect. I bet the most difficult part was defining a storage format and getting it integrated into an app built with the augmented reality SDK.

[Added in edit:]

Anyone on HN actively toying with a Kinect and want to share some video?

Something I'm doing with libfreenect: http://nitrogen.posterous.com/gesture-like-light-control-wit...

⬐ barefoot

Looks like you and I have toyed with the same idea (Kinect controlled lighting). I was shocked by how easy it was to put together and how effective it was. Probably not an easy thing to commercialize though ("yeah, just embed these sensors in your walls and replace all of your light switches...").

⬐ barefoot

I put together a very basic Kinect controlled lighting app for fun using ManagedNite:

http://www.youtube.com/watch?v=CF4mzdYo3VY

⬐ hammerdr

I have no video, but I was able to get a WebGL-powered game world running that was controlled through a Kinect using DepthJS. This was done in just over a day of playing around. Was pretty cool when it worked.

The source code is available on github at https://github.com/thoughtworks/kinect-spike

Edit: My co-workers were looking at me with concerned looks that day. I would stand up and wave my arms for 30 seconds and then sit back down at random times throughout the day. :)

⬐ rkalla

Very impressive Derek -- how easy was it to work with the Kinect data via the API? Is the information coming out of the sensors pretty straight forward or does it just come in as a video feed with additional metadata (positional?)

⬐ hammerdr

The way that DepthJS works is through three technologies: libfreenect[1], OpenCV[2] and then a simple Tornado (push) server. libfreenect is what grabs all of the data from the Kinect device and OpenCV is the 'interpreter'. This translates to just getting events such as 'move' in the browser.

[1] http://openkinect.org/wiki/Main_Page

[2] http://opencv.willowgarage.com/wiki/

Johnny Lee demonstrating Wii-based head tracking using the same plates-on-sticks imagery in 2007: http://www.youtube.com/watch?v=Jd3-eiid-Uw

Guy mounts two large framed LCDs in a wall and uses this effect to create a very effective virtual window which adjusts the view depending on viewer's position: http://www.engadget.com/2010/04/15/winscape-virtual-window-f... ("Winscape") in April 2010.

This particular implementation will probably never work, but that doesn't mean "3D movies" in some form will never work.

This guy built a stunning demonstration giving a 3d effect with a wiimote and a standard TV: http://www.youtube.com/watch?v=Jd3-eiid-Uw

What's ironic is that Microsoft recruited a high-profile Wii hacker to work on Kinect.

http://www.youtube.com/watch?v=Jd3-eiid-Uw

http://www.engadget.com/2009/06/12/johnny-chung-lee-joins-pr...

⬐ dzlobin

If I'm not mistaken he was already a microsoft research engineer

⬐ Uhhrrr

He was still at CMU when he was hacking Wii-motes: http://johnnylee.net/projects/

⬐ th0ma5

he also pioneered perhaps projection mapping

Specifically: http://www.youtube.com/watch?v=Jd3-eiid-Uw

Check out his entire youtube channel, it is full of interesting software/hardware hacks.

Here's some specific suggestions. I was most recently fascinated by this video on head tracking "virtual reality".

http://www.youtube.com/watch?v=Jd3-eiid-Uw

Another cool thing if you're into DIY robots (not really educational, but I found it inspiring) is the Yellow Drum Machine.

http://www.youtube.com/watch?v=_RyodnisVvU

Not sure why I got on the robot tangent but here's another good one, a 4-legged robot that's pretty good at navigating hills and obstacles. Especially the recovery from a fall sequence around 1:25.

http://www.youtube.com/watch?v=W1czBcnX1Ww

And here is hacking the Wii Fit controller and a Roomba to do a sort of vacuum-by-surfing... thing?

http://www.youtube.com/watch?v=tLbprdjTX0w&feature=relat...

⬐ asmosoinio

A very cool hack!

I wonder how widely used head tracking systems are for gaming? I saw one at my friends, who plays a lot of flight simulators on his PC, and I guess basically the technology was quite similar to the Wii remote - IR-leds and a camera.

⬐ samwise

Side note if you were to mount a camera on the hat/goggles and show the stream on VR goggles others could "share" the same experience.

⬐ euccastro

The same, minus control over your view. It's a very important difference. That's why you can get dizzy in a car when others drive, but not when you drive yourself.

Some people report motion sickness when using VR glasses; I think sharing the view would only make it worse for all but the guy in control.

⬐ samwise

I wish i was in "that" market because i would hire that guy in a heart beat.

Keep in mind the "magic" is in the software.

⬐ euccastro

The "magic" is in the man, but not in the software. I guess it only takes a knowledge of the low level Wii remote protocols and a good grasp of linear algebra to execute this. Most people that have that knowledge still don't come up with this coolest stuff.

⬐ hhm

The implementation for this is pretty simple. What this guy did is coming with a very original and interesting idea no other came before, and that was brilliant. I'm very, very impressed.

⬐ iamelgringo

Johnny Lee needs to get hired by someone.