HN Theater @HNTheaterMonth

When I saw this page a few years back I had an idea for a project. I want to create the lowest-latency typing terminal I possibly can, using an FPGA and an LED array. My initial results suggest that I can drive a 64x32 pixel LED array at 4.88kHz, for a roughly 0.2ms latency.

For the next step I want to make it capable of injecting artificial latency, and then do A/B testing to determine (1) the smallest amount of latency I can reliably perceive, and (2) the smallest amount of latency that actually bothers me.

This idea was also inspired by this work from Microsoft Research, where they do a similar experiment with touch screens: https://www.youtube.com/watch?v=vOvQCPLkPt4

⬐ lodi

If you ever end up doing this project, I find that sometimes it's hard to quantify if something is better when going in the normal-to-better direction, but it's always much easier to tell when something is worse when going in the normal-to-worse direction. So spend a few days or weeks getting totally acclimated to fast response times and then test if you can notice the difference with slow response times.

⬐ layer8

I like the idea, but note that (1) and (2) can depend on what you’re used to. The fact that one doesn’t notice a handicap doesn’t mean that there isn’t room for improvement, given some conditioning.

⬐ kaba0

I believe there are specific methods to negate these effects (something like going in a specific order with the values)

⬐ tcbawo

If would probably be interesting to randomize the latency for certain intervals with some kind of feedback mechanism to provide a blind study.

⬐ klysm

Sounds like a fun project to do, I wonder if you could even implement it in full discrete logic and skip the FPGA

⬐ DaiPlusPlus

Is there an appreciable practical lower-bound in latency to that? I’ve never understood how-and-why electronic signals can propagate down a wire so gosh-darn quickly: the speed of sound is what I’d have intuitively expected, not 50-99% the speed of light ( https://en.wikipedia.org/wiki/Speed_of_electricity )

⬐ skovati

Essentially, energy is carried throughout a circuit by the electric field created by a voltage differential, not by electrons pushing other electrons like how sound is transmitted through a medium. So signals propagate on the same order of magnitude as the speed of light. I think this is a relatively intuitive explanation: https://www.youtube.com/watch?v=C7tQJ42nGno

⬐ wincy

I remember when I got the iPhone X I was so used to higher latency that it felt like the iPhone was typing before I was. It was a very strange sensation until I got used to how quick it was.

https://www.youtube.com/watch?v=vOvQCPLkPt4&t=90s you can literally see the difference.

You can easily observe how significant latency is in videos like this: https://youtu.be/vOvQCPLkPt4?t=80 (Microsoft Research presenting its ultra low latency displays for touch interactions). Many mobile games have you drag and drop things, so it's not like it's just first person shooters that suffer from latency.

You're a lay person, you couldn't have known this, you're using words with very specific meaning to streaming (like latency) and you're comparing it to human reaction times, which are measuring something else entirely. You kind of reasoned about from a first principle in a very Paulgrahamarian way, and it led you deeply astray. That happens. And you're not the only person doing this, this is a comment section full of people who play games and parrot stuff they seen in YouTube, and don't have a concrete grasp of what it is they're even talking about, so it's understandable when it's laypeople shouting at laypeople that it's just a bunch of blah.

One of the reasons I hate HN and write in throwaways nowadays is that the comments section is a better example of Knoll's law than actual journalism.

⬐ anthonybsd

Wow. Condescending much?

⬐ gilrain

Appropriate in response to the breathtaking arrogance-in-ignorance of what it was responding to.

⬐ redox99

A bit condescending yes but he showed a really good example of how input lag is noticeable.

⬐ kbelder

This is the most factually true comment that I've ever downvoted.

⬐ Firmwarrior

That comment should be sent out as a blanket text message to everyone who commented on this post about latency IMO

The word needs to get out

⬐ Firmwarrior

Thanks for posting this. This comment section has been particularly frustrating to read, since it's a mirror of what I've seen in the real world. There are teams at big tech companies making TERRIBLE decisions about the future of gaming because they don't actually understand how latency affects games, and they aren't hardcore gamers so they can't feel the effects themselves.

Even the ~50ms total latency you get from locally streaming over a 1ms wired network (from buffering/inappropriate firmware design) ruins whole genres of high level gameplay. You miss tricky shots in FPS games, you can't confirm/link in fighters, etc.

I think you meant https://youtu.be/vOvQCPLkPt4 . It was indeed not too easy to find, due to SEO tactics polluting YT search with gaming-oriented stuff from recent times (where one may actually reach the frame rates necessary for such low latencies in a frame-by-frame display technology).

⬐ joshuak

Thanks! That is the one I was thinking of. Notice how even at 10ms of latency rubberbanding is easily visible.

Oh thank goodness, so it's not just me who is driven insane by the mouse lag in browser input.

And while I understand your cynicism, I don't think it's deliberate as much as it's complacency, given that it's latency issues all the way down when it comes to modern user interfaces.

A literal decade ago Microsoft Research uploaded a video highlighting how much of an effect minimizing latency for touchscreens had on the user experience[0]. The vast majority of tablets still utterly suck in this regard a decade later, and I wonder if any manage to get below the 10ms barrier.

And of course there is Dan Luu's famous 2017 article on computer latency in general[1].

[0] https://www.youtube.com/watch?v=vOvQCPLkPt4

[1] https://danluu.com/input-lag/

While the numbers may not be absolutely correct as stated in other comments, on a relative terms the conclusion from the trend is the same.

SSD was the biggest upgrade in performance in the past 15 years. There are still lots of system running on old HDD.

And the chart shows why I have been calling attention to latency computing for 10+ years. I hope the next stage is to reduce latency from Input, from keyboard to touch screen, and display input, and the actual display panel itself. On a 120hz screen even if you assume zero latency in everything else you still have a minimal of 8.3ms delay. Which is still quite obvious [1] once you put things in motion. And I think it is so hard there is enough room for at least another 10 years of hardware improvement.

[1] Applied Sciences Group: High Performance Touch

https://www.youtube.com/watch?v=vOvQCPLkPt4

>I just want to say I am glad pro gaming took over.

Yes! I have been crying about latency for nearly 10 years [1]. Computing has always been optimising for throughput. And before Pro Gaming, there just hasn't been a marketable incentive for companies to work on / minimise latency. Now we finally do!

Even in the best case scenario, the lowest latency is still 25ms, and in most cases we are still above 50ms. I think it is worth posting [2] Microsoft Research on Input latency. It would be nice if we could get average system end to end latency down to sub 10ms level. Which is what I hope work on VR will bring us.

[1] https://news.ycombinator.com/item?id=6422632

[2] https://www.youtube.com/watch?v=vOvQCPLkPt4

1ms is significant if you're a heavy user: https://www.youtube.com/watch?v=vOvQCPLkPt4.

⬐ amelius

That would require a real-time OS, I suppose.

⬐ jimktrains2

You only need a real-time os if you're guaranteeing maximum latency.

⬐ amelius

Which is exactly what was asked for.

⬐ gumby

“Realtime” doesn’t imply faster. In fact often the opposite. It just means certain timing constraints will be satisfied, often making explicit what the costs (tradeoffs) are.

⬐ amelius

Yes, the constraint here is 1ms.

⬐ perlancar3

From the video, 1ms feels real time, but with 5ms you can still notice the lag. So wouldn't it be more appropriate to say "4ms is significant"?

I wonder when can we get a Pen and Paper on a computer system with 1ms latency [1]? The Apple pencil said it has 9ms response time, which is already close to the lowest latency it could support on its 120Hz Screen ( 8.3ms per frame ).

And I may be in the minority but I much prefer there is an iPad that doesn't have a camera bump so it can lay flat on the table.

[1] https://www.youtube.com/watch?v=vOvQCPLkPt4

⬐ eloisant

Lower latency is always better, but in my experience even the 20ms latency of the Remarkable is something you get used to very quickly.

The video your linked is making latency even more obvious by moving the finger fast.

⬐ jlokier

Technically it is possible to update parts of some screens instead of the whole screen, so the screen refresh doesn't need to be the limit unless it's scrolling large areas.

⬐ nindalf

I got the iPad Pro because I want the lowest latency possible. But later I realised I can’t tell the difference between the iPad Pro (9ms) and iPad Air (18ms). I think other people might be able to tell the difference between 9ms and 5ms, but I doubt it’s distracting people out of their workflows.

That is what I have been banging on about for years. Latency added, not just in Software, but in Hardware as well from Display, Keyboard, Mouse and even Network. The past 20+ years the whole industry nearly went all out on optimising for throughput.

Sometimes I look at macOS, Windows 10. Apart from some UI facelift I dont even remember a single user feature that was relevant or important in the past 5 years. Apart from latency added cause of feature bloat

The beauty of Gaming is that there are finally some incentives for people to work on the issue.

And a video from Microsoft Research on 1ms Latency

https://www.youtube.com/watch?v=vOvQCPLkPt4

⬐ formerly_proven

> The past 20+ years the whole industry nearly went all out on optimising for throughput.

It used to be that GPU drivers would queue up to ~10 frames just to get slightly higher FPS in benchmarks (while adding basically a quarter of a second input lag)... naturally, this is an extremely bad trade-off for gaming, but made them look a few percent (at most) better on paper.

⬐ fouc

You reminded me of "Computer latency: 1977-2017" https://danluu.com/input-lag/

⬐ cheschire

"Finally" seems a bit myopic. The war against latency has been going on continuously since the beginning, even in gaming. Check out nearly any content John Carmack has ever written or spoken on the topic of gaming or VR, just as one example.

⬐ ectopod

For those of us who grew up with 8-bit home computers low latency was the starting point. On my first computer you could detect a keypress, update video memory and (assuming the beam was in the right place) see the results on screen in a few microseconds.

⬐ samplatt

>Sometimes I look at macOS, Windows 10. Apart from some UI facelift I dont even remember a single user feature

The user "feature" that comes quickest to mind for Windows 10 was the first run of calc.exe showing a popup notification begging for a 5-star rating in the Microsoft Store.

The fucking CALCULATOR. Literally the first and most basic function any computer ever had, which has had NO ui changes in the million software and hardware permutations for 70 years, because it's already perfect. Something has gone unfathomably, fundamentally wrong with how Microsoft works internally.

⬐ mongol

What rating did you give it?

⬐ samplatt

I gave it one star and left a scathing review. What are they going to do, not include a calculator in the next Windows?

⬐ noisy_boy

They can not include it and instead provide a menu option that actually takes you to the store on first run :)

⬐ rightbyte

Don't give them any ideas please.

⬐ ht_th

They might. It would not surprise me if a lot of people just type their calculations into the browser address bar or in their Internet search engine's input box. Or search for a calculator web application, and use that as their calculator.

⬐ alternatetwo

I installed the Win7 calculator, and it works like a charm. The no-hex functionality and splash screen really are terrible. I was used to start typing immediately after win+r calc ...

May be Speed isn't the right word, Latency would be better.

We can look at Input Lag [1], and Microsoft Research's work [2] on Touch Input. Apple's ProMotion being part of that as well. For the past 20 years we have make 10 - 100x improvement in bandwidth at the expense of Latency. Now we need to do more work on it . Especially if we want VR or AR which are extremely latency sensitive. John Carmack [3] used to talk a lot about it when he was still working on Oculus. How it was faster sending something hundreds of miles away than showing it on your computer screen.

[1] https://danluu.com/input-lag/

[2] https://www.youtube.com/watch?v=vOvQCPLkPt4

[3] https://danluu.com/latency-mitigation/

The 120hz screen is a big deal if you plan to pick up the pencil and use it for a lot of drawing/whiteboarding/notetaking. Latency really matters when you're drawing on the screen and can visibly see the line trail behind the pencil tip. Ideally, you'd want a 1khz screen for ultimate responsiveness [1], but that may be a long time coming.

[1] https://www.youtube.com/watch?v=vOvQCPLkPt4

True! I just watched this video [1] they put out with more details, and it sounds like they are working with monitor and mouse manufacturers to build true end-to-end latency measurement when using supported hardware. It would be easy for manufacturers to game these numbers, but hopefully Nvidia will provide some enforcement to prevent that. Maybe they could also add a microphone to measure audio latency because that is another huge problem with modern systems.

The video mentions that they get 15 ms end-to-end latency on a 360 Hz monitor in Fortnite. So that's what it takes for a modern system to finally beat the latency that you used to get on, say, an NES connected to a CRT. Still an order of magnitude above the limits of perception though [2].

[1] https://www.youtube.com/watch?v=-cXg7GQogAE

[2] https://www.youtube.com/watch?v=vOvQCPLkPt4

⬐ ksec

Yes! I watched that video many years ago. And then later I remember when media were hyping how "responsive" Apple Pencil was I was thinking what on earth are these people looking at. I thought to myself we would never get there because no one gives a damn about it. How the world has changed!

I often wonder at what cost, could we get it down to Sub Ms. Where everything from Input, Video and Audio are sync to the precision of 1ms. There used to be very little incentive to do so. Now that Gaming has taken over I hope it will develop some market perception and value so we could finally move into that direction.

The age of Instantaneous Computing.

You would definitely notice a 21ms lag while writing. Ideally you want to get below 10ms, but for physical-object-like responsiveness 1ms is the standard. See this old video from Microsoft research which demonstrates

https://www.youtube.com/watch?v=vOvQCPLkPt4

⬐ zaksoup

I just watched the video and maybe I missed it but doesn't that mean that in order to have 1ms latency you'd have to have a screen refresh rate of 1000 frames per second? That seems like it would cause serious cost increases for displays, controllers, and graphics cards.

⬐ the_pwner224

The 10ms demo is almost as good and much easier to accomplish, requiring only 100 fps instead of 1000. Actually you would need more than 1k fps - suppose it takes 0.5 ms for the touch sensors and CPU processing and graphics updates; then half the time after a touch you would miss the frame and have to wait 1 ms for the next display update.

To me 1ms doesn't seem worth it given the diminishing returns. Something like 5-10 would be way easier to do.

⬐ AshamedCaptain

Most devices these days actually heavily extrapolate, continuing drawing the line where they think you're going to draw it even before they've actually sensed where you drawed it. The effect is rather easy to see on a Surface Pro, but it actually works pretty well.

⬐ Zenbit_UX

You can cheat, Samsung was just bragging about this earlier in August when they showed off the new Note and smart pen. They used AI to predict where the user was likely to move the pen next to cut the perceived latency even further, some really clever stuff.

⬐ zaroth

If it helps, you just have to refresh the (very small number of) pixels under pressure, and only at such a high right for a brief moment while they are being pressed.

I wonder if you could have a separate layer which physically (chemically) responded to the pressure to make it look like the screen was drawing your line, but which only lasted for about 50ms after the pressure is removed.

This chemical layer would be visible for just enough time for the real eInk pixels to actually refresh underneath the "pressure mask".

⬐ setr

You'd be stuck with essentially one "material" though -- eg color or brush or whatever -- but I could see its utility as a dedicated single-purpose device (like only intended for sketching/notes -- which I guess is mostly true of remarkable anyways)

⬐ roughly

re: setr

> You'd be stuck with essentially one "material" though -- eg color or brush or whatever -- but I could see its utility as a dedicated single-purpose device (like only intended for sketching/notes -- which I guess is mostly true of remarkable anyways)

I'd be curious about human perception of color matching/etc at the <50ms level - I'd suspect if you had _something_ that then became the actual color/brush, our visual system would probably just backfill to say it'd always been that color/texture.

> That’s not true, though. It takes more than 30ms just to render your frame, deliver it to the monitor, and wait for a screen refresh.

For cheap non-CRT screens you are correct. Otherwise, not so much. And even if you're using a cheap screen, latencies of 50ms vs 30ms (typical numbers for vsync on vs vsync off) definitely have an impact on player performance besides just feeling more/less sluggish. See also my post here: https://news.ycombinator.com/item?id=23806493

> https://www.pcgamer.com/heres-how-stadias-input-lag-compares...

That PC gamer article is horrible and amateurish. There's no mention of what vsync/nvidia driver settings they used etc. Also games like Destiny are pretty far from being optimized for low input lag.

With CRTs and properly optimized/coded games like CS you can achieve input lag as low as 5ms (though that's just the time until the monitor begins drawing the first rows of pixels after the change).

Here's a few links on input lags measurements:

http://www.esreality.com/post/2691945/microsecond-input-lag-...

https://forums.blurbusters.com/viewtopic.php?f=10&t=1381&hil...

http://esreality.com/post/2640619/input-lag-tests-ql-csgo-q3...

People doing A/B tests:

https://forums.blurbusters.com/viewtopic.php?f=10&t=1134

And a demonstration that even input lag as low as 10ms can be noticeable:

https://www.youtube.com/watch?v=vOvQCPLkPt4

I'm taking issue with latency in general and it's dismissal. There are reams of ergonomic targets, data, and guides on user interface latency that are just completely ignored these days. A terminal is close to the simplest possible interface whose job is to provide input and output to a user with as little disruption as possible. 30 ms is in fact noticeable.

Here is a short video from Microsoft (about 10 years old now) on the experience of using a touchscreen with 100ms of delay vs 1ms showing the steps in between. The difference is extraordinary for the user even if the end result is the same. https://www.youtube.com/watch?v=vOvQCPLkPt4

⬐ PragmaticPulp

> Here is a short video from Microsoft (about 10 years old now) on the experience of using a touchscreen with 100ms of delay vs 1ms showing the steps in between. The difference is extraordinary for the user even if the end result is the same.

That’s additional input delay, not absolute latency.

More importantly, the visual effect you’re seeing isn’t the perception of input delay. It’s the positional difference between the finger location and the box location. Typing in a terminal doesn’t have the same visual difference. In fact, using a mouse doesn’t have the same visual difference because your finger isn’t right next to the screen for comparison. Try dragging a window across your screen right now. The “lag” is easily on the order of the 100ms shown in this demo, but you won’t find it anywhere near as disorienting because your finger isn’t on the screen for visual reference.

It’s not possible to have 1ms lag unless you have a 1000Hz screen and zero processing time for the inputs.

Compare that to the standard 60Hz displays that most laptops and cellular phones use. At 60Hz, the screen only updates once every 17ms. Add input lag, processing lag, rendering delay, buffering delay, and other delays and it’s nearly impossible to go from input to screen action in under 30ms. Most games are on the order of 60-80ms.

The individual pixels in your LCD can’t even toggle from black to white in 1ms.

A lot of the HN comments here are misunderstanding the time scales involved in all of this. For reference, a blink of an eye is about 100ms

You are probably thinking https://www.youtube.com/watch?v=vOvQCPLkPt4

Bottleneck is in the input layer. While using a desktop computer look at the mouse cursor, now move the mouse - hardware accelerated 2D graphics, imperceptible latency.

Low latency thru orthogonal multiplexing https://www.youtube.com/watch?v=t1VcC9_yhc0

iOS is smooth but not as smooth as I would like. The input-to-render latency matters a lot when dragging things and you can really feel it even though you probably wouldn’t notice the difference between 50 and 10ms of latency for something like a key press. See this video demonstrating what 1ms of latency looks like (and where they optimistically aim to have it in 10 years by 2022):

https://www.youtube.com/watch?v=vOvQCPLkPt4

100ms ping is much different than 100ms total latency.

Even locally, FPS games can reach upwards of 100ms latency locally even before hitting the network, so 100ms ping is more like 100+100=200ms of latency. http://renderingpipeline.com/2013/09/measuring-input-latency...

Additionally, we're extra sensitive to things like latency between a tablet pen drawing and it's stroke appearing, or rendering lagging behind VR HMD rotation, or audio latency if we're trying to jam. Even if we register ~100ms as "instantaneous" in a lot of scenarios, it can still cause troubling disconnects between action and result.

John Carmack on VR HMD latency: https://danluu.com/latency-mitigation/

John Carmack noting that depressing a 360 gamepad trigger can take 20ms: https://twitter.com/id_aa_carmack/status/233240260568551425

A Microsoft Research video on tablet input latency, where the effects of 100ms vs 1ms latency are clearly visible: https://www.youtube.com/watch?v=vOvQCPLkPt4

Just watching their promo video about the Astropad Studio on their website, what I see is the latency for drawing is annoyingly bad (and they show it exactly when they're talking about "lightning fast speed" :O )

https://vimeo.com/198755916

But part of the problem is probably the drawing software too, because even native drawing has different latency between drawing apps:

https://www.youtube.com/watch?v=zNbvYaO0ZZ0

It's not an easy problem because even 1 ms latency is noticeable in drawing as Microsoft research showed:

https://www.youtube.com/watch?v=vOvQCPLkPt4

⬐ cma

Apple OS updates broke low latency even on things that only send the mouse, like Synergy (power saving causes latency when idle and wakes up for responsiveness only on hardware input, or something like that). My guess is sidecar uses OS level stuff to get around some of those limitations, which may have affected astropad too?

It can be very detectable, I'd say. Try watching this: https://www.youtube.com/watch?v=vOvQCPLkPt4 It shows the difference between from >100ms to ~1ms. It's mind baffling.

⬐ ksec

I wonder how did they manage it. Even the Screen itself has a 1 - 4ms latency. That is excluding the Input latency and Software in between.

The threshold for noticeable touch "rubberbanding" is 1 ms:

https://www.youtube.com/watch?time_continue=52&v=vOvQCPLkPt4

I'm happy to see someone thinking about latency about like I do. This Microsoft study (https://youtu.be/vOvQCPLkPt4) keeps lingering in the back of my mind.

The most annoying latencies I've seen so far are scrolling on Android and Trackpad mouse moving on Linux (x11).

⬐ visarga

Yep. That's the same video I was thinking of.

⬐ watmough

Outlook 2016 is the pits. Microsoft should pay attention to their own research.

Yes, and something like a touch interface is very obvious, too.

Look at this video on Youtube from Microsoft Applied Sciences Group: High Performance Touch.

https://www.youtube.com/watch?v=vOvQCPLkPt4

You can plainly see that 100ms is an eternity. While keyboards aren't the same thing, but high latency is noticeable. I had to change a keyboard because of the near eternity of when I pressed a key and when it registered on the screen. The difference between the old and the new is quite stark.

⬐ clebio

You realize this video (or a version of it) is linked in the article itself, yeah?

Yes there is. Humans are normally not very responsive to the time between when an action is started and when it is carried out on the screen, but this changes for actions like drawing and dragging. This is the reason for the new Pro Motion behavior on the recently released iPad Pro models. For more information, this video [1] is a good place to start.

[1] https://www.youtube.com/watch?v=vOvQCPLkPt4

⬐ njarboe

"not very responsive"? I'm responsive. I mutter things like, "Shit, why does it seem that the amount of lag in my computer's response to my inputs get worse and worse as its physical components get faster and faster."

⬐ dyladan

I think you misunderstood what I meant by non-responsive. 10ms latency from click to action is nowhere near as noticeable as 10ms latency when dragging, because the mouse visibly lags behind.

Microsoft Research showed that anything above 1ms latency is noticeable on touchscreens: https://www.youtube.com/watch?v=vOvQCPLkPt4

End-to-end latency is definitely something that deserves attention. It'd be interesting to modify mouse&kb by adding LEDs in series with the switches. That way you'd get good visual indicator when the event happened without any added latency. Combine that with 144hz monitor and actual high-speed camera (I hear good things about Chronos camera), and you could do very accurate software latency measurements

This MSR video from few years back is also pretty cool wrt touch interface latency: https://www.youtube.com/watch?v=vOvQCPLkPt4

Agreed, 100ms feels very different from 50ms or 10ms.

Microsoft Research has an interesting video[1] on the UX impact of different touch latencies. Even below 100ms, users are very sensitive to latency.

[1] https://youtu.be/vOvQCPLkPt4?t=55

Seems the holy grail is a really tough 1ms: https://youtu.be/vOvQCPLkPt4?t=52s

Hard to see from the videos, but I don't really expect anything better than 10ms. At least, we can hope to see incremental improvements in the coming years.

Millisecond-level feedback is incredibly exciting. It's only at these rates that the limits of human perception seem to no longer detect the disparity between the real world and digital interaction.

This strongly reminds me of a video[1] from Microsoft Research a few years ago in which touch-based interaction was demonstrated at 1ms latency. It's surprisingly more realistic than even the 10ms level.

[1] https://youtu.be/vOvQCPLkPt4?t=52s

⬐ zokier

That video is from 2012 and represents 100ms as typical at that time. I wonder if we have made any progress in the three years that have passed. Does anyone know what the touch latency of typical Android device/application would be?

I'd be curious to also know what sort of keyboard/mouse to display latency PCs typically have. I've heard some talk about display input latency, but I don't remember hearing anything about e.g. keyboard latency or OS latency (in this context)

⬐ Qantourisc

Your link is one of the main reasons I use actual paper to write on.

⬐ None

None

⬐ Mithaldu

As much as i agree that latency reigns the human mind supreme, i find that video just a little bit disingenuous, since they compare vector drawing with bitmap drawing.

⬐ leni536

What about drawing in bitmap with low latency and building the vector drawing from buffered user input in the background? Then replace bitmap with vector when it's available. It could be low latency and vector drawing at the same time.

⬐ rasz_pl

>only at these rates that the limits of human perception seem to no longer detect the disparity

IF you use the brute force approach of vomiting pictures in general direction of an eye.

Human vision system is incredibly meager and clever at the same time. Your eye doesnt really capture whole picture all at once, you have a huge blind spot in the middle + part of your nose obscures vision, you can only see sharp shapes in the center and fast movement on the boundaries, data stream consists of shaky blurry fragmented mess. Its the brain that filters and glues it all together into coherent picture.

Example - go to a mirror and look into your eye. Now find another person and look into their eye, you will be surprised to see yours was steady, and theirs is all over the place. Now check out your blind spot http://io9.com/5804116/why-every-human-has-a-blind-spot---an...

Saccades produce a lot of blurry artefacts that are simply thrown away. I read somewhere brain ignores about 2-3 hours of visual data a day.

We already have incredibly fast micromirrors, I wonder if someone is working on a microprojector that tracks saccades and displays only part of the picture eye is currently fixated at. This would allow constructing high resolution scenes using lower resolution projector. 120 Fov requires >500mpixels, but you can only see ~7mpixels at a time. Quick back of napkin calculation tells be you could bump perceived resolution of Occulus by ~40x

⬐ random_passerby

Moreover seems like our perception of time and space do change during a saccadic eye movement.

At least according to that paper from ten years ago : http://www.nature.com/neuro/journal/v8/n7/abs/nn1488.html

Maybe is there more recent researches in that field.

⬐ None

None

⬐ jay-saint

  " I read somewhere brain ignores about 2-3 hours of visual data a day."

I imagine that is mostly due to PowerPoint and meaningless dashboards.

Reminds me of the 1ms touch delay technology also from Microsoft's applied sciences group.

https://www.youtube.com/watch?v=vOvQCPLkPt4

You'd be surprised how sensitive we are to latency on input. Microsoft themselves did studies on it: https://www.youtube.com/watch?v=vOvQCPLkPt4

Microsoft demoed 1ms touchscreen lag in 2012.

Most modern touchscreens still have 100ms lag.

https://www.youtube.com/watch?v=vOvQCPLkPt4

⬐ jcl

Nice video! It really shows the perceptual difference well.

However, I'm not sure it's fair to call it a "touchscreen" when it's most likely a touchpad input surface aligned with an overhead projector. It's impossible to say from the video how feasible it would be to achieve the same performance with an actual touchscreen, let alone a high-resolution mobile device. But I'm looking forward to it. :)

⬐ wingerlang

I was under the impression that that setup was something including a camera or something, and not an actual touchscreen.

⬐ i_am_ralpht

Yeah, though maybe you could do it with FTIR instead of capacitive, so you wouldn't need external cameras.

⬐ rasz_pl

This lag is in the software stack (mostly in UI graphics). Not in hardware.

Remember Androids ~>200ms sound lag?

⬐ imaginenore

Definitely not. Most of mobile software strives to run at 60fps. That's just 16.7ms.

⬐ rasz_pl

display animation at 60fps and react to a hardware input through layers and layers of software abstraction are two different things.

Run of the mill 3M controller has 5ms latency

http://datasheet.octopart.com/87-5961-211-3M-datasheet-21189...

5 year old Fujitsu FMA1127 had 10ms.

http://www.fujitsu.com/downloads/MICRO/fme/tsc/FMA1127_Data_...

Interrupt latency is measured in nano seconds.

⬐ imaginenore

If you had read the PDF, 5ms is the "minimum touch time", which is not latency, but a minimum touch time it can register.

The same software has no problem rendering complex games and doing all kinds of game logic, all in 16.7ms. What makes you think asking for a touch input will take 15 times longer?

⬐ rasz_pl

Actually its response time.

I already gave you an example of software stack impact on latency - Android sound lag went down from >250ms to respectable 10ms for some Nexus devices. Blame buffers.

⬐ imaginenore

> Actually its response time.

Actually, no, it isn't.

⬐ rasz_pl

"This controller is well known for its < 5.4 milliseconds response time"

⬐ kalleboo

What I've read is most of the lag in the more optimized touchscreen stacks (read=iOS) is due to filtering and smoothing, which takes a few samples to do. The worse stacks (early android) had poor drivers and a lot of layers of abstraction, or worse, Java (maybe this is what you mean when you say "UI Graphics", but the raw numbers we're talking about here are on a far lower layer than scrolling in lists).

Graphics rendering lag is at worst 2 frames of video (double-buffering).

⬐ footpath

The creators of the drawing app "Colors!" for 3DS and iOS also tested their app's latency across a wide range of devices, and they achieved a 9ms latency in 2004's Nintendo DS:

http://www.collectingsmiles.com/news/measuring-latency-in-co...

  Nintendo 3DS – 23 ms
  PlayStation Vita – 49 ms
  Surface Pro – 100 ms
  iPhone 5 – 81 ms
  Galaxy Note – 71 ms
  Galaxy S3 – 104 ms & Nexus 10 – 99 ms
  Galaxy Note 2 – 132 ms
  Wii U GamePad – 53 ms
  Nintendo DS – 9 ms

To get an idea of the impact and importance of touch latency, see this intriguing demo by the Microsoft Applied Sciences Group: https://youtube.com/watch?v=vOvQCPLkPt4.

Drawing is one of those things that has felt awful on iOS devices to me.

⬐ coldtea

>Drawing is one of those things that has felt awful on iOS devices to me.

Compared to what, since, according to TFA they have the smallest touch latency?

Perhaps you haven't tried the right apps. Most drawing apps (including major names, like Autodesk's) have slow-ass drawing code. Heck, some painting apps are even slow on my iMac (Corel Painter, ArtRage).

That's not because of "touch latency" though. It's because of slow draw engines. And I say that, because I've seen apps with very fast responsiveness.

Try Procreate ( http://procreate.si/ ), which uses the fastest engine I've seen (specially coded in OpenGL and 3D-accelerated). And check the artwork created with it by some of the community users (there are 2-3 videos on their site it's AMAZING).

Two other apps I found fast (but not as fast) are: Ideas, by Adobe, and Paper.

⬐ archagon

As much as I love my iOS devices, they still have MASSIVE latency compared to my Wacom tablet. There's just no comparison. I guess the touch processing had a lot of overhead.

⬐ eropple

Same here. My 21UX is staggeringly more responsive, and I expect the newer ones are better still.

⬐ coldtea

No, they use different technology for touch registration.

(And, for what is worth, older Wacom tablets were also slow).

⬐ mdwrigh2

Touch processing adds somewhere in the ballpark of 15-45ms of latency depending on the technology used in the screen and the quality of filtering in the touch controller. Anything about ~20ms of latency (total from touch to display) is perceivable by humans, is usually a good approximation for noticable increases or decreases in latency (i.e. shaving off 2ms doesn't really matter, but shaving off 15-20 will be noticeable).

⬐ kllrnohj

> Compared to what, since, according to TFA they have the smallest touch latency?

Drawing apps can easily end up with significantly higher latency than the baseline if they aren't careful with how they do input smoothing and such. It's really easy to get that wrong and bloat your touch latency.

⬐ kmfrk

    Compared to what

Paper, dude(tte)! The analogue, lowercase-letter kind. :)

Paper and Procreate are excellent apps, but the input lag is very noticeable on my iPad 3; if you move your finger really fast and stop, you can see that it takes a short amount of time, before the drawn line catches your finger.

It wasn't long ago that dead-tree books were preferable to tablets, before Retina was a thing. And you could still argue print books are preferable.

Sometimes, the best Technology can achieve is the bar set by the non-digital world, and until then, it's going to feel grating to pedantic curmudgeons like yours truly. :)

+++

EDIT: A closing thought. The highest bar is always human perception:

(1) The optimal FPS for a videogame is, to my knowledge, 60.

(2) The optimal display resolution for a reading device is, TMK, ~326 PPI (Retina).

That is the goal. Anything lower than that will stick out and annoy people like me. Anything higher is not for the engineers, but the marketing department.

⬐ plorkyeran

60 fps isn't optimal - 120hz displays have a small but noticeable benefit when the rest of the hardware is good enough to lock in 120 fps.

⬐ lukeschlather

have you tried the microsoft surface pro?

⬐ kmfrk

Nope - I haven't even seen one!

⬐ vidarh

The "catches your finger" thing may be input lag, but it much more often is amateur level coding, it seems to me.

One of the first things you learn if you try to do a simple drawing app on a slow machine (think Commodore 64 or similar early 80's 1-2MHz computer), is that if you try to process every position change, you will lag badly.

First rule of doing a decent paint app, is to decide how large deltas you can accept between each position where you actually draw, and drop events accordingly. Small delta, and you will lag; high deltas and you either need to draw lines or will get "dots" instead of a continuous line, but your lag will be limited to that of the input device. Even an early 80's home computer can give you "lag free" painting this way, at the expense of precision if you do large, very fast movements over the screen.

This is "paint app writing 101". Yet a lesson that seems to have been lost on most people writing paint apps these days, possibly because they've become accustomed to computers fast enough that they're no longer constrained by it.

I don't know about iOS, but I do know from observation that out of 30+ draw/paint apps I've tried on Android while trying to find one I'd actually be happy to use (I did not find any; I'm picky), it is obvious that this is the problem for the vast majority of them for the simple reason that there's a huge spread in observable lag for them, and so even if we assume the fastest of them are limited by input lag, the extent of the lag on the majority of them will then still be down to crappy coding.

(the quality of paint apps for Android is just beyond awful in general; don't know about iOS)

⬐ neonkiwi

I've done some research which involved the perceptual effects of latency, so I was interested in the test rig from their video and found a paper they published in ACM about it: http://edgey.com/wp-content/uploads/2013/01/p453-ng.pdf

Turns out that to achieve this 1ms latency system (once you have that, it's simple to add artificial latency) they had an FPGA directly processing touch data from a resistive sensor over a 2 MBps serial (optical) connection, computing the center of mass and directly driving a DLP projection display so that parallax between the surface being touched and the display would be eliminated. Nice rig!

⬐ joeblau

That video was a great. Thanks for sharing it.

⬐ kmfrk

It's one of my favourite technology videos/demos of all time.

Every time someone criticizes Microsoft, I remember this video, knowing Microsoft's failures aren't for lack of talent.

⬐ theon144

Sadly, Microsoft's research group and just about everyone else at Microsoft aren't the same people at all :\

⬐ mistercow

Of course the video is a little bit misleading when it talks about 1ms refresh, since touch response stops being the limiting factor before that point. AFAIK the fastest refresh rate for any phone on the market is 250Hz. That's 4ms refresh, which is great, but of course you also have to factor in time to actually process the touch event, change the image, and render it.

⬐ rorrr2

If we want a 60Hz app, the touch latency + processing + display shouldn't exceed 16.7ms.

1ms touch latency would be nice indeed, but I think 5-8ms would work just as well on most devices.

⬐ nrivadeneira

It's not misleading at all. They specifically said it was a test setup and stated that their aim is to be at around 1ms latency in the next decade. Obviously that means working on the other limiting factors to get there as well. It's not like they'll just try to get the touch response to 1ms and ignore screen refresh rates.

⬐ Pxtl

Yep. There's a decent homebrew paint app on the Nintendo DS that runs circles around the smartphone counterparts... The Ds has a joke of a processor, terrible resolution, and no memory worth mentioning. But it has a resistive pressure-sensitive touchscreen and a stylus. That's enough to make a huge difference. Resistive touchscreens are fast and pressure sensitive, allowing far better conditions for art.

⬐ aikinai

The sad thing is that official games aren't allowed to support the pressure-sensitivity provided by the DS hardware, so it's only used in homebrew apps like the paint app you mentioned.

⬐ MBCook

Do you know why that is? For example, does the pressure-sensitivy vary wildly between individual screens?

⬐ Pxtl

Incredibly so. Calibrating the aforementioned paint app is a huge bitch. But it's great afterwards.

⬐ archagon

I remember reading somewhere that the DS homebrew "pressure sensitivity" was merely a hack. Since you hit more neighboring pixels when you push the stylus down, you get more "pressure". (This could be outdated information, though.)

Even 25–50ms is noticeable in certain situations. See http://www.youtube.com/watch?v=vOvQCPLkPt4 for an illustration of how things get noticeably better, even as you go below 10ms.

⬐ kpeatt

In situations like this it's definitely important to get that response lag down as much as possible — you're looking at direct 1:1 control with that though which is slightly different than a tap. Like you said, it all depends on the situation.

Thanks! I didn't realize so much latency came from the device itself, but it's good to see confirming evidence that decreasing it would tremendously improve the experience.

That's worthwhile research to measure how much improvement makes a difference (i.e. until it becomes imperceptible), but unfortunately I get the impression they are only measuring it, not solving it. I gather their test setup is faked, just to enable measurement, because of the careful way he expresses it (I expect it doesn't actually use touch). I hate to reference a youtube comment for support, but the top one says it's IR + projector http://www.youtube.com/watch?v=vOvQCPLkPt4 And other links to that "Applied Sciences" Group don't mention any on-going research on it (or even the guy in the video)... http://research.microsoft.com/en-us/groups/asc/default.aspx Searching ms research didn't up any other mention of the two researchers involved (Albert Ng and Paul Dietz) - Albert's now at Stanford, and Paul seems to have many projects.

100ms to 1ms is a long way...

From TFA:

    > The first important idea is that in Wayland, every 
    > frame is regarded as "perfect." That is, the client 
    > application draws it in a completed form [...]

This is interesting, but also seems to run counter to recent work on the importance of very low latency feedback. See John Carmack's recent article on latency mitigation for VR headsets[1][2] for a very detailed example of this issue. Similar ideas also apply to touch[3][4] and even good old keyboard & mouse input schemes. There's a tradeoff between "perfect" frames and latency (see Carmack's post). From the quote, it appears that Wayland's design choice may limit achievable latencies for low-latency applications.

[1] http://www.altdevblogaday.com/2013/02/22/latency-mitigation-...

[2] #1 on HN: https://news.ycombinator.com/item?id=5265513

[3] https://dl.acm.org/citation.cfm?id=2380174

[4] https://www.youtube.com/watch?v=vOvQCPLkPt4

⬐ nona

https://plus.google.com/112891790407866414198/posts/LditLoiD...

Check the last two comments.

⬐ rayiner

Low latency feedback is important for some things, but complete double buffering is the right default for a window system, because flicker is even worse than touch lag.

This is one of the reasons people think iOS is so "snappy." It goes to great lengths to never show you a partially-drawn frame. If you compare Safari to say Internet Explorer on Windows RT, you see that IE looks like ass when rendering complex pages, because it'll happily show you frames it's done laying out yet.

⬐ thechut

From the video comments:

"Furthermore these aren't even touchscreens. This is just a projector with IR sensor allowing us to experience 1ms but is in no way a touchscreen. The purpose of the video is to show us what touchscreens will be like with 1ms delay"

Anyone happen to know if it uses their new ultra-responsive (1ms) touchscreen technology? I remember being pretty excited about that when they demoed it, but it seemed like it wouldn't be rolled out for a while.

http://www.youtube.com/watch?v=vOvQCPLkPt4

This is the demo video you're talking about : http://www.youtube.com/watch?v=vOvQCPLkPt4

⬐ artursapek

Ah yes, I remember this. Worth noting that his device is a projector putting the image on a screen, and I think it's got a lot more computing power behind it than an iPad has available.

⬐ Lerc

The current latency isn't a CPU issue though. I'm not sure of the refresh rate of the iPad (60Hz?) but the hardware has no trouble updating the display in a single frame.

It should be noted that unless Microsoft had a 1000Hz display running, there was something fishy going on with their 1ms update.

⬐ chadv

It's a custom display purpose built for this demo. I believe that it is 1000Hz.

⬐ owenfi

From this high speed video I took I think the refresh rate is 60Hz (this shows it is at least 60).

http://owenimholte.wordpress.com/2012/03/30/ipad-3-screen-re...

The app looks really, really cool, but I think this input lag demonstration by Microsoft has really spoilt me in the same way that I can no longer look at mobile devices without Retina and not cringe.

https://www.youtube.com/watch?v=vOvQCPLkPt4

⬐ reedlaw

What are the chances that real hardware response will get down to the 1ms range in the near future? I'm thinking that each interface, from the touch pad to the CPU to the display, will introduce significant lag.

Related to your point. Microsoft Research and some prototype displays.

http://www.youtube.com/watch?feature=player_embedded&v=v...

⬐ ekianjo

Good point, that is exactly what I had in mind when I wrote my comment.