Performance

One of the main questions we get asked about Frisbee, is how does it compare to NATS or GRPC? We've spent a lot of time trying to address this concern properly, and we're excited to report that Frisbee's performance is heavily optimized for performance and efficiency - regardless of what you're using it for.

Please Note: These Benchmarks are Open Source and available in their own Github Repository. Instructions on how to run these tests on your own machines are available in the repository's readme.

We encourage everyone to run these benchmarks for themselves.

NATS Benchmarks

We've gone through many iterations of this benchmark to make sure that we're doing an apples-to-apples comparison.

NATS is designed for PUB/SUB applications, and it's heavily optimized to do exactly that. Frisbee on the other hand, is designed for custom messaging paradigms that may be very dissimilar to PUB/SUB. So, in order for this benchmark to be fair to NATS, we had to implement PUB/SUB behavior using Frisbee, and used that for the benchmark.

Benchmarking Architecture

Both the NATS and Frisbee benchmarks are designed to do the following:

  1. The Broker starts up and listens for connections
  2. A Receiver connects to the Broker and Subscribes to a given topic (TOPIC 1)
  3. A Sender connects to the Broker and starts sending N messages on TOPIC 1
  4. Once the Sender has sent N messages, it sends an END message on TOPIC 1 and waits to receive a message on TOPIC 2
  5. When the Receiver receives an END message on TOPIC 1, it sends a message on TOPIC 2 signaling that it has received all N messages
  6. The Sender received a message on TOPIC 2, it calculates the amount of time it took for N messages to propagate through the broker
  7. The Sender repeats steps 3-6 and averages the calculated times

For NATS, we're using the official NATS Broker, along with nc.Publish and nc.Subscribe to handle the pubs and subs.

For Frisbee, we've written a naive implementation of a broker that saves subscribed Frisbee connections in a map and forwards published messages to them in a loop.

These benchmarks were run 100 times each and then averaged.

Benchmarking Results (1P1C)

We ran these benchmarks using 1 Producer and 1 Consumer on Digital Ocean's CPU-Optimized Droplets with 32vCPUs and 64gb of Ram. We provisioned 3 VMs, running FreeBSD 12.2 (with ZFS), and used the private VPC network.

There are a few important observations to make here. First, the throughput trends are very similar between the NATS and Frisbee benchmarks - this tells us that our benchmark implementations are indeed very similar. The second important thing to note here, is that our throughput for Frisbee is slightly higher than that of NATS - even though NATS is explicitly designed for PUB/SUB implementations.

We're able to achieve these results without any optimizations within our sender, receiver, or broker because the Frisbee read and write loops are already extremely efficient.

Benchmarking Results (2P2C)

Another question we wanted to answer about NATS is how it scales with multiple producers and multiple consumers versus our Frisbee implementation.

For this benchmark we provisioned 5 VMs, each one of Digital Ocean's CPU-Optimized Droplets with 32vCPUs and 64gb of Ram, running FreeBSD 12.2 (with ZFS), and using the private VPC network.

We set it up with 1 broker for both Frisbee and nats, and 2 producers publishing messages on two separate topics, and two consumers subscribed to each of the separate topics. We then ran the benchmarks concurrently and averaged the results.

As seen in the 1P1C benchmark, the throughput trends are very similar. This solidifies our assumption that our benchmark implementations were similarly in their design. The second important thing to note here, is that our throughput for Frisbee is again slightly higher than that of NATS, telling us that Frisbee scales just as well as NATS does, if not better.

GRPC Benchmarks

Coming Soon

Last Updated: 2021-06-08