HN Theater @HNTheaterMonth

> no big player took to developing cool things on top.

Just one data-point, but WhatsApp engineering was adamant that FreeBSD was a key precursor that allowed them to scale 2m+ connections per host on their (rather beefy) boxes: https://youtu.be/TneLO5TdW_M

> It just shows you that having a technically good solution is irrelevant.

True. I've seen many promising projects die/stagnate at the hands of unpopularity, *BSDs wouldn't be the first. Sigh.

You should be able to do this by just expanding some sysctl limits on FreeBSD 10+. https://www.youtube.com/watch?v=TneLO5TdW_M

It's required because your phone is where your messages are stored.

Whatsapp don't retain messages/media after they've been delivered to your phone, which is a compelling privacy feature for many.

It's also what allows them to serve such an enormous user base with limited hardware. Their technology stack (FreeBSD/Erlang) is pretty interesting, more info here:

2014 talks by Rick Reed:

https://www.youtube.com/watch?v=c12cYAUTXXs

https://www.youtube.com/watch?v=TneLO5TdW_M

Slides:

http://www.erlang-factory.com/static/upload/media/1394350183...

There's likely no technical reason why you couldn't use a pc instead of a phone for users that want to use the pc as the primary client (with the phone optionally accessing the DB on the pc in the same way that the desktop client does for the phone). Perhaps they've decided that this is a small and declining market.

Edit:

Slides for second talk

http://www.slideshare.net/iXsystems/rick-reed-600-m-unsuspec...

440k connections/sec, 1.1 million msgs/sec, 1 billion images/day, and that was in 2014...

⬐ executesorder66

> It's required because your phone is where your messages are stored.

So are you saying that a desktop PC's hard drive can't handle storage of some text messages but a phone can?

⬐ yxlx

No, he's saying that in order for the messages to be available, they have to be stored locally and since most people need access to the messages from their phones, it makes sense to do it this way to ensure that all messages are stored on the phone, instead of users ending up with some messages on their phones and some on their PCs.

⬐ xomateix

Of course, he is not saying that, he is just explaining how whatsapp is actually storing the messages.

⬐ Longhanks

Neither does iMessage store messages on servers, and yet I receive iMessages on every device I signed on.

⬐ kccqzy

But you can't retrieve historical iMessages on a new device.

⬐ hudell

I'm not really asking for that feature. If I could send and receive messages on the computer without having to open the smartphone app every five minutes for it to restore the connection, I would be happy already;

⬐ uola

"Perhaps they've decided that this is a small and declining market."

For consumers (and facebook is a consumer company) it's all about who owns mobile (and can also compete with facebook). They want desktop to just be enough of a feature to be more appealing than other platforms, but not enough that it detracts from mobile.

⬐ Freak_NL

> There's likely no technical reason why you couldn't use a pc instead of a phone […]

Any personal computer built in the last five years can do anything a smartphone or tablet can in terms of processing power and connectivity. A smartphone is a computer with hardware that enables it to use cell phone networks and make calls.

My inner cynic strongly suspects that Facebook and other similar corporations really like the control they have on the overall user experience on the two major mobile operating systems; i.e., eyeballs on a smartphone or tablet are worth more than those on a general purpose computing device.

Too much freedom on a personal computer; with browsers that feature all kind of privacy enhancing add-ons such as ad-blockers and tracker-blockers. Much harder to monetize.

⬐ kome

You are totally right! That's the obvious answer. On desktop you have too much freedom.

⬐ rhaps0dy

I'm going to assume this is sarcasm and upvote.

I can personally guarantee to you from direct current experience that there is nothing particularly solved about running services for even 50 million users.

Go watch this: https://www.youtube.com/watch?v=TneLO5TdW_M#

There was a talk + slides about WhatsApp FreeBSD usage [1, 2].

At around 23:50 in the video, Rick Reed talks about slide #17, they say they are running 9.1 - 9.3, and looking at 10.1 (not in a hurry as 9 works for them).

[1] https://www.youtube.com/watch?v=TneLO5TdW_M

[2] http://www.slideshare.net/iXsystems/rick-reed-600-m-unsuspec...

⬐ barkingcat

-CURRENT is not a single version number - it refers to a development branch. If someone says CURRENT is version xx, then they don't really know what they are talking about - because while CURRENT may become a versioned release down the line, that is very different from what -CURRENT represents (which is a named tree in source control that people develop against)

All of those freebsd versions you refer to (9, 10) are part of the production release cycles and thus are not -CURRENT.

If whatsapp uses 10.1, then they are NOT using -CURRENT

⬐ WestCoastJustin

Yeah, I think we are both thinking the same thing. I put the links, and version numbers, in support of tptacek's thought that they were not using -CURRENT.

⬐ perlgeek

This feels like mixing of levels to me. The Internet traffic of nearly every user will run through a Cisco or Juniper router. Does that make us all IOS or JunOS users?

⬐ rdtsc

It will and if Junos or ios stop working so will the internet to a large extent. It speaks to reliability and scalability aspects.

You could say the same about erlang for example, it used by WhatsApp and in many internet-cell network gateways.

⬐ justincormack

Netflix is the other FreeBSD based service that many people dont know about. While the control plane is the much hyped Linux on AWS stuff, the actual streams are FreeBSD boxes in ISPs[1].

[1] https://openconnect.itp.netflix.com/software/index.html

⬐ tachion

There's also Juniper, EMC2, Apple, Yahoo (back in the day, at least), Sony PlayStation and many, many others.

⬐ adamnemecek

Also HN :-)

⬐ justincormack

I would only really count Juniper in that list, building on and with FreeBSD. Apple take some bits but their product is not really FreeBSD (alas) and they dont seem to give back. AFAIK EMC and Yahoo are ex-users now, or largely.

⬐ feld

EMC Isilon still uses FreeBSD. There are employees who are also FreeBSD developers. They're working hard on giving back changes so they can jump to a much newer FreeBSD release.

Yahoo's FreeBSD isn't dead; YBSD! still exists too.

⬐ praveenster

Since Whatsapp doesn't have client facing web page, but only api's used by the client apps, are their scalability requirements different from those that have a web interface as well?

⬐ None

None

⬐ Demiurge

messages are probably much smaller, but more numerous and latency has to be 'realtime', but also client-server-client juggling. so, to me this means they need to respond faster, more often, and deal with more sharing than a website.

⬐ jambo

I found the pragmatism and sense of deferring until the least responsible moment—especially the speaker's view of what that moment is—very interesting.

In the Q&A "What do you use for configuration management?" "<slight laugh> `make`", going on to say they don't have enough node types or a large enough cluster to do automation yet.

⬐ general_failure

Not sure the title is correct. After all, the people use WhatsApp through Linux (Android), Symbian or whatever and not FreeBSD.

⬐ adamnemecek

The servers were (are?) FreeBSD.

⬐ andor

Yes, of course, and Whatsapp users care very much about what's going on behind the scenes ;-) I'm not a mainframe user because I book flights, or a Windows user because i tried Bing.

⬐ tachion

It takes very little sense of humor to detect the fact that the title is a nerdy joke ;)

⬐ tachion

The servers are FreeBSD 9.x. After Facebook acquisition they've tried migrating it all to Linux but quickly discovered that Linux cant cope with that amount of traffic and they decided the FreeBSD is the tool to be used here. That lead to 'famous' Facebook job ad looking for Linux kernel engineers to help them bring the Linux TCP/IP capabilities in par with FreeBSD. Currently they're testing FreeBSD 10.1-RELEASE for their next platform OS upgrade.

⬐ agumonkey

Pretty telling technical story and also flattering for the FreeBSD team.

⬐ ps0ps

This is not true. Facebook has never attempted to migrate them to Linux. If WhatsApp wants to convert to Linux, it is their decision. More than likely they will stay on FreeBSD as that is what they know how to run and a good way to kill a product is to force them to do things instead of focusing on building their product.

⬐ sandGorgon

http://bsd.slashdot.org/story/14/08/06/1731218/facebook-seek...

⬐ toast0

A job posting in August indicates a failed migration attempt that couldn't have started until October, because that's when the transaction closed?

⬐ wslh

> Linux cant cope with that amount of traffic

I like FreeBSD but I don't think we can say that Linux can't cope with that amount of traffic while Google and Facebook use Linux. There is a missing piece here that's not about network traffic.

⬐ justincormack

I think it is amount of traffic per server. WhatsApp didnt have a lot of servers and maybe reengineering for massive scale out on Linux doesnt make sense versus staying with what works.

⬐ nonane

Are there more details available about this? What specifically about the Linux TCP/IP stack is weaker than BSD's stack? Where does the weakness lie?

⬐ jlouis

It doesn't have to be a weaker stack per se. It is also a lot about knowledge. WhatsApp vested much time into understanding FreeBSD. It may be if you vested the equal amount of time on Linux with knowledgable people, then you would see roughly the same performance.

If I should take a guess, the advantage of the FreeBSD stack is that it is simpler and easier to understand. Hence, it is easier to tune for performance.

⬐ atmosx

Do you have any idea why Facebook would rather develop a TCP/IP Stack for Linux to match FreeBSD's, instead of further evolving FreeBSD's stack? Licensing issues?

⬐ justincormack

How could licensing issues be a problem?

I think people prefer a monoculture, even though the security idea of having multiple software stacks is appealing, thoroughly testing issues on two OS stacks is a lot more work, especially at Facebook scale.

Presumably they are tied into Linux for a lot of their internal infrastructure.

⬐ Perdition

The main security threat these days isn't viruses or worms so a monoculture makes no real difference.

In fact running multiple operating systems is more likely to lead to a misconfiguration issue which allows an attacker into your network.

⬐ justincormack

If someone getting into your network is your threat model then, well, you have already been owned. If you want to have resilient services where you can swap them out if there are security issues, a monoculture is not a win.

⬐ sciurus

Facebook is almost certainly further evolving Linux's existing TCP/IP stack, not developing a new one.

⬐ MasterScrat

I would rather guess it is for consistency reasons. People are more familiar with Linux than FreeBSD.

Similarly one of the reasons Facebook relies on PHP is because a lot of developers are familiar with it.

⬐ justincormack

I know lots of people who are more familiar with FreeBSD who are working on Linux systems. The Linux monoculture is just massively dominant.

⬐ tachion

While I have no actual knowledge why they made that decision, its not very hard to guess, and the possible answers are around their own staff familiarity with FreeBSD, the pros of having the platform simplified by using single OS (Linux in their case, since for sure they've more servers with Linux already than they've acquired with WhatsApp) and all possible security and financial implications following these factors. However, it seems like at least for know they might be staying with FreeBSD a bit longer, what's a great PR/recognition win for the FreeBSD itself. I really doubt its for 'licencing issues' because (religious wars aside) the BSD license is way more attractive to businesses than GPL one.

Said all that - I might be terribly wrong about everything I said, because its not an insider knowledge ;)

⬐ sandGorgon

Correct me if I'm wrong, but one of the biggest scaleups in FreeBSD seemed to be zero-copy sockets. This should be solved in Linux when kdbus is turned on (replacing the current IPC : D-Bus).

⬐ glasz

just throwing myself in here: Starve [1] also runs on FreeBSD.

[1] https://www.starve.io

⬐ sandGorgon

if one HAD to choose a non-erlang stack (for whatever reason), what would you choose ?

Go, Nodejs or Java - I'm mainly concerned about clustering, because I think that is what gave the Whatsapp devs their ability to scale(without having to do too much re-engineering).

Or can the clustering be moved to something like the queuing layer ?

⬐ pjmlp

Java, of course.

There are lots of JVM implementations to choose from, many with high quality code generation (both AOT and JIT) and GC implementations.

It has VM and cluster monitoring tools only comparable by .NET offerings.

The majority of big data solutions are built on top of it.

There are lots of libraries to choose from.

If Java 8, the language, isn't your thing, there are lots of modern languages to choose from.

⬐ sandGorgon

I wsa hoping you wouldnt say that :(

Java high performance clustering is... intimidating. It is too enterprisey. From my (little) research, it seems that one would need to build this ground up on top of Netty or Undertow.

⬐ marktangotango

Care to elaborate? Load balancing to a bunch of http servers, sticky sessions, or store the session to a db, or the best option is no session data, memcached, its all pretty straight forward?

⬐ dschiptsov

Showing FreeBSD/Erlang stuff that just works and cope with substantial amounts of traffic is not politically correct, because it will cause a severe pain of cognitive dissonance to the Docker/JVM/Node/Cloud crowds..

Lets chant together: JVM is the best VM in the world with millions of man-hours invested in optimizations. Servlet/JSP are mature, battle-tested, enterprise-grade, type-safe solutions. NodeJS is the only way to develop high-scalable internet services. Everything must be a containerized micro-service, orchestrated by Docker/CoreOS/systemd.

⬐ pjmlp

It is worse than that, where you would offshore development like on the other cases? :)

⬐ wallflower

To me, the more fascinating fact is that Erlang enabled their massive ability to scale.

http://highscalability.com/blog/2014/2/26/the-whatsapp-archi...

⬐ rudiger

Not just Erlang/BEAM optimization, but the team's background in high performance, C-based systems on FreeBSD enabled them to achieve massive scale.

⬐ Ixiaus

What's more fascinating is that erlang underpins many more staples of the internet than most geeks even know. WhatsApp is just one 19B example.

⬐ pjmlp

Not only the Internet.

A few German game studios are using it on their backends.

⬐ marktangotango

Physics in Erlang? Game scripting engine implemented in Erlang? What types of games are we talking about here?

⬐ pjmlp

I wrote backend. Physics and game scripting belong on the client side, usually.

http://de.slideshare.net/wooga/erlang-the-big-switch-in-soci...

http://sdtimes.com/enterprises-can-learn-from-game-developme...

⬐ jambo

Yep. Especially when you consider that things like RabbitMQ, Riak, ejabberd, etc. are written in Erlang and take advantage of OTP.

⬐ None

None

⬐ dschiptsov

The "miracle" of Erlang/OTP is that it is well-researched and based on proper concepts, so there is absolutely no usual bullshitting here.

The crucial mantra by Joe Armstrong is "to model a behavior in a "parallel" real world one have to choice a proper abstraction (which is about granularity, not classes or objects), then everything follows".

So they have selected a connection as a fundamental abstractions and a lightweight process as a unit of granularity. Everything else follows, indeed.

Processes share nothing, so there is no locks, semaphores, race-conditions. There is even no loops - only recursive procedures, which could be restarted if crashed, or even its code could be replaced without stopping the service (why, a new copy of the code would be called "in a next recursive call", because it is stateless/pure-functional).

Each lightweight Erlang process serves its own connection, so it is perfectly natural and efficient to block/sleep on it (the OS kernel will do the job). It is not a "busy waiting", and there are no callbacks either. No "event buses", no nonsense.

There is no threads, so no shared state or even shared stack. If a process crashes (why should it? - it communicates with outside world only via message-passing) it cannot corrupt or lock any shared data, etc. The whole class of problems which comes form a flawed concept of pthreads does not arise.

There is no "reactive asynchronous event processing (written in Java or Scala)", there is no non-blocking I/O (btw, the only real non-blocking I/O could be done by an OS - aio_read/write and friends). There is no meaningless over-abstraction. That is why it works so well. Mo magic.

⬐ Moyamo

I wouldn't say these are BSD users. If I am a windows user and connect to google's linux servers to search, does that make me a linux user?

⬐ mace

Agreed. I think this title is a bit flippant and I'm sure the 600M+ WhatsApp users don't care what the servers are running.

The important takeway is that FreeBSD and Erlang have worked really well for WhatsApp allowing them to grow their userbase insanely fast while still staying relatively small.