HN Theater @HNTheaterMonth

I was not the who asked but thanks for the reply jerf, indeed I thought you had in mind more complex cases than the GenServer one! It is not that sync is hard in Erlang/Elixir but once you want to involve multiple processes, it may be the case.

For example, in pipeline parallelism, we don't want to be completely sync between the stages, because it would mean you would always have at most N "events" in the pipeline at any moment (where N is the number of stages). If you want to have some sort of bound, you need to devise a message protocol between the stages. That's what the folks behind Reactive Streams are doing with their back-pressure work.

This is exactly the kind of situations I want to make easier in future Elixir versions. I talked about them from the perspective of collections (and not FBP) in my Elixirconf keynote: https://youtu.be/EaP0y4pdKD0?t=1846. We are experimenting with the addition of a GenRouter that would specify how processes communicate (with back-pressure) eventually supporting custom dispatch rules. We are also looking into projects like Basho's sidejob (https://github.com/basho/sidejob) and Ulf's jobs (https://github.com/uwiger/jobs) to ensure we can also provide sampling based routing.

On top of that, we need to discuss data-based parallelism (where we split/spread the data once instead of passing it around) and how it would all fit with supervision trees. It is a lot of work and we likely won't be able to solve all cases but I am very excited about the possibilities if we get it right.

⬐ jerf

I would encourage you to consider whether there may be more useful primitives that the Erlang VM could support for this case, if you aren't doing that already, which I can't think of how to quickly check for so apologies if I'm late to that party. Hopefully you'll have more pull on the matter than Some Schmoe like me would. :) A lot of these things strike me as either inefficient or even infeasible to emulate via strict user-level Erlang-style message passing, but are really easy with some well-chosen primitives added in, even if they are only exposed very carefully at the Elixir and Erlang levels.

Some of this is personal development bias, I have to admit, in that I don't really "believe" in APIs that try to abstract away the difference between local and network traffic. Making the difference small may be advantageous, but I actually don't like it when it's reduced to zero. Usually, of course, a language goes the direction of making local traffic easy and network traffic unbelievably hard, Erlang makes network traffic easy, but then makes it somewhat more difficult to know whether you're going across the network or not. That's really useful for a lot of things, but sometimes if you're willing to agree that you won't need the network stuff you can get some big wins; in Erlang-land, for instance, consider the way larger binaries are shared instead of copied, an optimization that works locally and is mostly meaningless across a network.

⬐ josevalim

> I would encourage you to consider whether there may be more useful primitives that the Erlang VM could support for this case, if you aren't doing that already

It is a chicken-egg issue though. Before proposing anything like this to the OTP team, I need to prove it is useful and have folks build something relevant with it. So I need to make the best with the abstractions we have today and then make a case.

For this reason, a lot of the challenge will be in optimizing the "topologies" to avoid copying as much as possible. Data-based algorithms for parallelism (farm, pmap, etc) will likely be more useful and that would be the next milestone.

> Some of this is personal development bias, I have to admit, in that I don't really "believe" in APIs that try to abstract away the difference between local and network traffic.

That's a very good point. I was also warned by Konrad from the Akka team that, if we rely on ack messages for back-pressure, they likely won't work on a distributed setup as messages have no delivery guarantees. This means we wouldn't be able to abstract the network anyway.