So how does Linux compare now with FreeBSD in terms of throughput and latency? I remember like 10 years ago Linux had issues with throughput, which is why Netflix went with FreeBSD. Are they similar now?
So, this is an honest question. What kind of performance do Linux based CDNs get out of a single box?
At Netflix, we can serve over 90Gb/s of 100% TLS encrypted traffic using a single-socket E5-2697A and Mellanox (or Chelsio) 100GbE NICs using software crypto (Intel ISA-L). This is distributed across tens of thousands of connections, and all is done in-kernel, using our "ssl sendfile". Eg, no dpdk, no crypo accelerators.
I'm working on a tech blog about the changes we've needed to make to the FreeBSD kernel to get this kind of performance (and working on getting some of them in shape to upstream).
It doesn't hurt to get a performance boost from your processor's crypto instructions, assuming you optimized your cipher lists to prefer crypto with a modern hw implementation (AES128-NI is 179% faster than RC4).
But is this traffic ongoing connections, new connections, a mix? They have different penalties, and result in different numbers: 90Gbps of ongoing connections might be, like, 100,000hps, but 90Gbps of new connections during primetime might only net you 50,000hps. And are you using Google's UDP TLS stuff?
Google also hacked on the kernel a lot to improve their performance, I don't know if any of that's upstream currently though. Maybe Cloudflare can answer you, as they seem to support the most HTTPS wizardry of the big CDNs.
That would be the part of Netflix not running on AWS. :-)
So the library feed can serve tens of thousands of streams, an aggregate 100Gbps, on a single node. And then... how many nodes to support the front-end UI operations to get to that point?
It's funny how amazingly efficient we can be moving encrypted bits, but to support the APIs for login, browsing titles, updating account info, and setting up a stream; I'm going to guess ~100 of those nodes for every one of your stream-tanks?
> So how does Linux compare now with FreeBSD in terms of throughput and latency? I remember like 10 years ago Linux had issues with throughput, which is why Netflix went with FreeBSD. Are they similar now?
I believe the open connect team at Netflix choose FreeBSD because a lot of them used to work at Yahoo and had lot's of FreeBSD experience. Not so much because of a performance difference between the two. As for now, the two network stacks are pretty equal when it comes to performance, some work loads are better on FreeBSD some are better on Linux.
Yea no problem, I'm having a hard time finding all the articles/ benchmarks I remember reading about this subject. But this paper has some useful info[1]. Basically FreeBSD is good for throughput. In the paper I linked to you can see FreeBSD has a higher throughput than Linux, but you can also see that FreeBSD is using more CPU than Linux is. Also Linux generally has lower latency than FreeBSD, which makes sense because Linux is used extensively by high frequency trading firms. However there are still HFT firms using FreeBSD.
I'll edit this post as I find the other articles, videos and benchmarks about this subject.
Edit: I don't really care for Phoronix benchmarks but here's some benchmarks showing Linux winning some and FreeBSD winning some benchmarks[2].
I also like this talk by a George Neville-Neil a FreeBSD networking engineer. Start at 57:45 [1]. That part of the talk is about differences in how the two different Network stacks are implemented.
