HN Theater @HNTheaterMonth

Queue theory suggests the more nodes you add to handle your queue then the longer your queue can survive (facing rising traffic) but the faster it will collapse when it finally runs out of capacity. Zach Tellman went over this in one of his talks and he had a nice graphic to demonstrate what this looks like:

https://www.youtube.com/watch?v=1bNOO3xxMc0

I suppose any complex system has dynamics like that, more capacity to absorb load, but then a faster collapse when capacity is surpassed.

Hemmingway's Laws Of Motion applies:

”How did you go bankrupt?” Bill asked.

”Two ways,” Mike said. ”Gradually and then suddenly.”

Put a bit differently by the economist Rudiger Dornbusch:

”The crisis takes a much longer time coming than you think, and then it happens much faster than you would have thought.”

lifted from:

https://conversableeconomist.com/2015/01/17/the-hemingway-la...

Yeah this is true, typically you want to keep blocking functions outside this. By default, core.async allocates as many threads as cores in your CPU.

In addition to this, I personally find core.async to deal poorly with flow control (e.g. slow subscribers slowing down the entire flow for every subscriber), and seems to not have a lot of telemetry / it’s difficult to find out what’s going on in the thread pool behind it.

I’ve personally settled on Zach Tellman’s Manifold library, which provides a much more sane abstraction and is fully compatible with core.async.

For some addition thought and material on these topics, I highly recommend watching this talk by Zach, “Everything Will Flow” https://youtu.be/1bNOO3xxMc0

⬐ jwr

Hmm, interesting — but manifold is clj-only, and I use core.async both on server side and client-side in ClojureScript.

I like this essay but I think it is very strange that he doesn't link to the talk that Zach Tellman gave in which Tellman talks about the origins of Factual's Durable Queues:

https://www.youtube.com/watch?v=1bNOO3xxMc0

Factual's Durable Queues are a fairly minimal queue effort that is useful in cases where you need some minimal way of handling back pressure, because a service downstream has failed. In his talk, Tellman talks about the fact that he needed to write a lot of data to AWS S3, and sometimes S3 stops accepting writes, so he needed an easy re-try mechanism.

I used Durable Queues everywhere in my code and find it very handy. It's useful when you need a very minimal queue that still has the backing of a file system.

⬐ emidln

durable-queue was one of my favorite pieces of Clojure kit.

No. Check out this video by Zach Tellman. He talks about queues and how they break down under load. One of the least intuitive things he points out is that when you have more processors, the breakdown tends to be more of a step function: everything is running smoothly till the moment that it isn't.

https://www.youtube.com/watch?v=1bNOO3xxMc0

The point he makes arises from basic queue theory and is applicable to all kinds of systems, and how those systems react to load. It's got little to due with particular hardware and everything to do with basic math.

⬐ hackits

That was a great talk and really sound recommendations he makes.

On dealing with backpressure, I would recommend what Zach Tellman writes about in his video "Everything will flow":

https://www.youtube.com/watch?v=1bNOO3xxMc0

Tellman also tells his own adventure about a system something like this:

"a system that would latch up into an overload state because in overload it would log errors at a severity high enough to force the log file to be flushed on every line"

He talks about some problems he faced when writing to Amazon S3, when S3 stopped accepting requests. It's a fascinating talk.

Anything by Zach Tellman:

https://github.com/ztellman/aleph

https://github.com/ztellman/automat

In terms of his ideas, watch "Always be composing":

https://www.youtube.com/watch?v=3oQTSP4FngY

And he offers some great thoughts about queues and backpressure in "Everything will flow":

https://www.youtube.com/watch?v=1bNOO3xxMc0

⬐ slantedview

I was going to post this. Not only is Zach's work very creative, but it's very well written, composable stuff.

⬐ escherize

Wow, I knew about aleph, but just learned of automat. I was always a fan of FSMs in school, and automat's readme gave me goosebumps. Thanks!

⬐ dschiptsov

> "Always be composing"

the ideas were stolen from the Wizards talk

http://ocw.mit.edu/courses/electrical-engineering-and-comput...

⬐ prospero

I wasn't aware of this talk, so I'm not sure how I could have stolen anything from it. Cool link, though, thanks.

⬐ dschiptsov

I should have used quotes.

See also a very nice video from ClojureWest about queues in system architectures: https://www.youtube.com/watch?v=1bNOO3xxMc0

The queue are everywhere - your messaging queue, the threadpool, the hardware threads, and other layers of the stack and APIs you use. The video adds the interesting detail that as you add more tellers (workers) you learn of impending disaster only in the outlier p99 (or higher) latencies; by the time your p85 latency rises, you're already about to stall out.