We’re announcing fRPC, an RPC framework designed from the ground up to be lightweight, extensible, and extremely performant.

fRPC: A Faster, More Flexible RPC Framework

We're moving our community from Slack to Discord, but... why? We'll answer all of your questions and more!

Building a New Community

As an open source company, we're big on sharing _everything_. We figured that should apply to our hiring process, too!

Hiring at Loophole


Today we’re announcing [fRPC](https://frpc.io), an RPC framework that's designed from the ground up to be lightweight, extensible, and extremely performant. We built fRPC because we loved the idea of defining our message types in a standardized proto3 format and having the protobuf compiler generate all the necessary glue code for us - but we didn't like the overhead of encoding and decoding messages in the protobuf format. Instead, we needed a wire protocol that was lighter and faster than HTTP/2.

fRPC offers a few major improvements over existing RPC frameworks like gRPC:

- **Speed** - On average fRPC outperforms other RPC frameworks by 2-4x in an apples-to-apples comparison, and is easily able to handle more than 2 million RPCs/second on a single server
- **Flexibility** - Not only does fRPC allow developers to deviate from the standard request/reply messaging pattern and implement custom patterns alongside their existing RPCs, but developers also have the ability to turn fRPC off and retrieve the underlying TCP connections so they can be reused for something else
- **Familiarity** - Using fRPC feels very familiar to anyone who’s used gRPC before, which means that developers can take advantage of the speed and extensibility that fRPC provides without a steep learning curve

fRPC works with the proto3 syntax, meaning developers can bring their existing proto3 files and tooling. Under the hood, fRPC uses a completely custom messaging format and generates a highly-optimized client/server implementation - the result is an RPC framework that [substantially outperforms and outscales existing frameworks like gRPC](#the-f-stands-for-fast).

fRPC is available now in early alpha under the Apache 2 open source license, and documentation is available on [https://frpc.io](https://frpc.io). We’ll also be releasing fRPC for Rust and TypeScript soon, with more languages to follow!

So what does the "f" in fRPC stand for?

## The “f” stands for “fast”

If you’re looking for an RPC framework that can easily handle 2M+ messages/sec, we think you’ll find fRPC to be exactly what you’re looking for.

But exactly how fast is fRPC? Well, we originally designed fRPC for our own messaging needs at [Loophole Labs](https://loopholelabs.io/),
where we needed to send both large and small amounts of data in a latency-sensitive manner. We also needed it to be massively scalable,
able to handle thousands of concurrent connections and able to service millions of messages.

I won’t go into how we achieved this here, but we are working on a series of engineering-focused blog posts that will talk in depth about how fRPC works under the hood. In the meantime,
we recommend you check out the [fRPC Github Repository](https://github.com/loopholelabs/frpc-go), or take a look at some of the [generated fRPC code](https://github.com/loopholelabs/frpc-go-benchmarks/blob/master/frpc/benchmark/benchmark.frpc.go) - we made sure the code generated by the protoc compiler is easy to read and understand.

Now for some benchmarks! We can’t just claim that fRPC is faster than a battle-tested tool like gRPC without backing it up with an apples-to-apples comparison. These benchmarks are publicly available at [https://github.com/loopholelabs/frpc-go-benchmarks](https://github.com/loopholelabs/frpc-go-benchmarks), and we encourage you to run them for yourselves.

To make sure our benchmark is fair, we’ll be using the exact same proto3 file as the input for both fRPC and gRPC. Moreover, we’ll be using the exact same service implementation for both the gRPC and fRPC servers - the generated service interfaces in fRPC are designed to look the same as in gRPC, so using the same service implementation is extremely simple.

### Benchmark Proto3 File:

```protobuf
syntax = "proto3";

option go_package = "/";

service BenchmarkService {
  rpc Benchmark(Request) returns (Response);
}

message Request {
  string Message = 1;
}

message Response{
  string Message = 1;
}
```

We’ll be running 2 different benchmarks with an increasing number of concurrent clients to show off both the throughput and the scalability of fRPC when compared to gRPC.

It should be noted that the benchmark service itself simply echoes data back to the client without doing any work - the point of the benchmark is to show off the throughput and scalability of fRPC, not the performance of the service itself. That being said,
we've also created benchmarks that do perform work and those are available in both the [publicly available benchmark repository](https://github.com/loopholelabs/frpc-go-benchmarks), and on our documentation site, [https://frpc.io](https://frpc.io).

The following benchmarks were performed on a bare metal host running Debian 11, 2x AMD EPYC 7643 48-Core CPUs, and 256GB of DDR4 memory. The benchmarks were performed over a local network to avoid inconsistencies due to network latency.

### Client Throughput

![Client Throughput](/blog/images/2022-08-05-announcing-frpc/32byte.png)

In this benchmark each client will repeatedly make RPCs to the fRPC or gRPC server using a randomly generated 32-byte message, and then wait for a response.

In each run we’re increasing the number of concurrently connected clients and we’re measuring the average throughput of each client to see how well fRPC and gRPC scale.

From the graph above it’s obvious that fRPC consistently outperforms and outscales gRPC - often by more than 2x. In the case of 8192 connected clients, fRPC’s throughput is still 112 RPCs/second while gRPC drops to only 29.

That means that clients using fRPC get almost 4x more throughput than gRPC using the same services and the same proto3 files.

### Server Throughput

![Server Throughput](/blog/images/2022-08-05-announcing-frpc/throughput.png)

Now let’s look at how fRPC servers scale as we increase the number of connected clients. For this benchmark, we’re going to make it so that each client repeatedly sends 10 concurrent RPCs in order to saturate the underlying TCP connections and the accompanying RPC server.

As before, we can see that fRPC consistently outperforms gRPC - but as we increase the number of clients it’s also clear that fRPC does not get as slowed down as the gRPC server does. It’s able to handle more than 2,000,000 RPCs/second and the bottleneck actually seems to be our bare metal host as opposed to fRPC.

In the case where we have 8192 connected clients, we can see that the gRPC server is able to handle just less than 500,000 RPCs/second - whereas fRPC can handle more than 4x that.

These benchmarks show off just a small portion of fRPCs capabilities, and we encourage everyone to run these for themselves. We’ll also have similar benchmarks for other popular RPC Frameworks available on the [fRPC docs site](https://frpc.io), as well as benchmarks comparing fRPCs messaging format with protobuf and other serialization frameworks.

## The “f” stands for “flexible”

Performance is great, but it’s not the only reason we built fRPC. One of the main requirements our team had that led to the creation of fRPC was the ability to do things that just weren’t possible with gRPC.

Because of its architecture, developers actually have the ability to turn fRPC off, and retrieve the underlying TCP connections so they can be reused for something else.

Within our team this feature has been extremely popular because it’s allowed us to establish some fRPC clients, send a couple RPCs for authentication, and then turn off fRPC and use the existing TCP connections directly with an etcd or PostgreSQL client!

What’s more, fRPC also allows developers to deviate from the standard Request/Reply pattern that RPC is so famous for. It’s now actually possible to implement custom messaging patterns alongside your RPCs using the same framework, same code base - and the same clients.

Want to reuse your servers as message brokers for a low-latency Pub/Sub system? How about reusing clients to stream metrics in real-time (while also making standard RPCs)?

Frankly, we have no idea how developers will take advantage of these features - but within our engineering team it’s already been described like “taking off handcuffs we didn’t even know we had”.

With fRPC it’s now possible to implement custom messaging systems alongside standard RPCs - something that would normally require multiple frameworks and a host of niche knowledge to achieve.

## The “f” stands for “familiar”

Since it’s conception, what’s always drawn developers to gRPC is the ability to define message types and handlers in the proto3 format - and have the protobuf compiler automatically generate the client and server code in whatever language is required.

The idea was always to make cross-service (and more importantly, cross-language) communication a breeze. Developers flocked to gRPC simply because it was a reliable way of doing that.

With fRPC our goal was to deliver a high-performance RPC framework where developers could take advantage of its speed and extensibility without a steep learning curve. We achieved that by building an experience that feels very familiar to anyone who’s used gRPC in the past.

As an example, here’s a simple “Echo” service (written in Go) implemented in both gRPC and fRPC:

### gRPC Implementation:

```go
package main

import (
	"context"
	"os"
	"net"

	"github.com/loopholelabs/frpc-go-examples/grpc/echo"
	"google.golang.org/grpc"
)

type svc struct {
	echo.UnimplementedEchoServiceServer
}

func (s *svc) Echo(_ context.Context, req *echo.Request) (*echo.Response, error) {
	res := new(echo.Response)
	res.Message = req.Message
	return res, nil
}

func main() {
	lis, err := net.Listen("tcp", os.Args[1])
	if err != nil {
		panic(err)
	}

	server := grpc.NewServer()
	echo.RegisterEchoServiceServer(server, new(svc))

	err = server.Serve(lis)
	if err != nil {
		panic(err)
	}
}
```

### fRPC Implementation

```go
package main

import (
	"context"
	"os"

	"github.com/loopholelabs/frpc-go-examples/frpc/echo"
)

type svc struct{}

func (s *svc) Echo(_ context.Context, req *echo.Request) (*echo.Response, error) {
	res := new(echo.Response)
	res.Message = req.Message
	return res, nil
}

func main() {
	server, err := echo.NewServer(new(svc), nil, nil)
	if err != nil {
		panic(err)
	}

	err = server.Start(os.Args[1])
	if err != nil {
		panic(err)
	}
}
```

See the similarities? The service interface is exactly the same, and we can use the exact same implementation of the `Echo` method for both!

## What does the “f” actually stand for?

It actually stands for Frisbee, the proverbial wizard behind the curtain. When we started out at Loophole, we realized we needed a messaging framework that was effortlessly extendable while also being able to scale to handle thousands of concurrent connections on a single node.

[NATS](https://nats.io) was the first solution that came to mind, but we realized it wouldn't work for our needs because we needed to do much more than just Request/Reply and Pub/Sub - and we needed much more control over the route that messages took than what NATS could provide.

So, we set about building Frisbee - a messaging framework designed to implement other messaging frameworks, something that would handle all the plumbing for us and let our developers focus on the actual messaging logic instead.

It's important to note the distinction between fRPC and Frisbee. fRPC uses proto3 files to generate client and server implementations that use Frisbee under the hood. This is why fRPC is so performant compared to other RPC frameworks - the Frisbee messaging framework and wire protocol are lightweight and extremely optimized.

At its core, **Frisbee** is best described as a **bring-your-own-protocol** messaging framework, and the goal was to make it possible for developers to define their **own** messaging patterns and protocols, but have the actual lower-level implementations done for them by Frisbee.

With Frisbee you can implement any protocol or pattern you'd like, but since the RPC pattern is so common, fRPC allows you to generate the necessary client and server code for RPCs very quickly with nothing more than a proto3 file.

## What’s Next?

This is an initial release of fRPC for the Go programming language, and our goal for the ecosystem is to make it possible to use fRPC across multiple languages. Rust and Typescript are first on the docket, but more languages will follow depending on the needs of our community.

We also don’t support all of the features that gRPC does yet, most notable being Streaming and OneOf message types. Rest assured, these are actively being worked on and we’d love for contributors to help out in making these available as quickly as possible.

We also plan on growing the capabilities of Frisbee itself, not just by implementing it in other languages, but also by continuing to improve its performance and ease-of-use. We’d also like to implement an in-browser version of Frisbee that relies on WebSockets - this will make it possible to use fRPC directly in the browser without needing to modify your backend at all.

## Get Started!

Check out our [getting started guide](https://frpc.io/getting-started) to quickly get up and running with fRPC! We’d love to hear what you think about it and we encourage folks to contribute to both [the documentation](https://frpc.io)
for fRPC as well as [the project](https://github.com/loopholelabs/frpc-go) itself by making a pull request!
If something isn’t working right, please feel free to open an issue on our [Github](https://github.com/loopholelabs/frpc-go).

If you need help getting started, the **Frisbee** Channel in our [Discord](/discord) is a great place get help with all
things Frisbee and fRPC! You can also follow us on [Twitter](https://twitter.com/LoopholeLabs) to stay up to date on all things Loophole!

<cta/>



We've worked really, _really_ hard over the past few months to design a hiring process that we're proud of! Of course we expect the process to evolve over time, but for now we want to share some thoughts about the way we've set things up, why we've done it this way, and also I'll probably rant about the broken state of the tech industry interviews. Let's goooooo!

_**Note:** Our process is designed specifically for interviewing software engineering candidates since that's all we're hiring for at the moment. We're hoping to reuse most of what we've built when we come to hiring others, but we absolutely expect to revamp huge chunks of it to better assess candidates with different skillsets._

# Our Design Principles

To build a great hiring process, we believe it's necessary to know what matters most. For us, we discovered that our most core value is that we care _a lot_ about making the process as frictionless as possible for candidates. While iterating on our process, we developed the following principles which have helped guide us towards the process we have today.

## Principle 1 - Transparency Is an Absolute Requirement

We've tried to make our interview process as transparent as possible. For example, some companies provide vague timelines around when you'll hear from them, but most leave candidates in limbo, wondering if they should send a follow up email. We find that to be _completely unacceptable_, so we have strict SLAs on responses and we communicate those timelines to candidates. Here are a few other examples:

* We publish salary ranges (including equity) in every job description. We don't want anybody to waste their time applying if we can't meet their financial needs.
* Candidates receive an agenda beforehand with every question we'll ask during the interview. It's an opportunity for us (meaning both the company _and the candidate_) to decide if we want to work together. It's not a pop quiz.
* We will **always** let candidates know whether or not we're moving forward as soon as we can. A candidate should never have to wonder whether we're taking our time to decide or if we've already moved on.

## Principle 2 - The Candidate's Time Is Valuable

Our time as interviewers is also valuable, obviously, but the candidates aren't being paid for the time they spend on our process. We recognise this, so we try to ensure that we're as efficient as we can be with their time, while also requiring very little extraneous work from the candidate. To this end we have...

* ...eliminated cover letters. We don't want 'em. Send us see a resume (or LinkedIn, or Github, or website... we're not picky) and let's get that screening call scheduled.
* ...set temporal boundaries on our interviews. Each section of the interview is designed to fit within a specific amount of time. We're flexible — some sections take more or less time depending on the candidate — but we have an absolute time limit on the interview process. If we haven't asked all of our questions before we get to the time limit, no worries. We prioritise our questions to get to the most important ones first, and we will never hold a candidate longer than the scheduled time limit (currently 1.5 hours).
* ...automated scheduling (mostly). We use automation tools that allow candidates to schedule their interviews without requiring a bunch of back–and–forth over email. As long as there's time on the appropriate calendars, scheduling should be as quick as the candidate can make it.

One more thing that we do: schedule our post-interview meetings as soon as we can. Often we'll have them immediately after an interview. Our guarantee is that no candidate will have to wait more than 5 business days for a response, and sometimes we can make same day offers!

## Principle 3 - Assume Competency

Tech interviews are all over the board, but there's one portion that's basically standard across the industry: programming challenges. They come in many forms:

* Take home projects
* Shoulder surfing disguised as pair programming
* Whiteboarding questions

Additionally, adjudication of these challenges is inconsistent. Very few companies have a legitimate, non-biased scoring system for their challenges. Most have no heuristics at all, instead relying on subjective opinions from the interview team. These challenges almost always fail to represent a candidate's capability. They're not representative of what candidates do on a day-to-day basis, and they introduce a significant amount of stress and anxiety for most candidates.

Instead of leveraging this terrible, archaic, flawed methodology, we decided to take a different approach. Rather than requiring candidates to perform a circus act for us we just... assume they know what they're talking about. 😱

I know, I know... it's a wild concept, right? The idea that people applying to engineering roles _actually understand engineering?_ We subscribe to the idea that we should trust the people applying to have the expertise they claim to have.

On a more serious note, the benefits of this have been substantial. We do still ask candidates to submit an open source project that they've worked on (there's an alternative option, too), but the goal isn't for us to adjudicate their technical chops. I'll discuss that more in a bit.

# The Actual Process

I'd be lying if I said it hasn't been hard to escape from the trappings of the "standard" interview process. With these principles in place, however, we've been able to design a process that we think allows us to make smart hires without forcing the candidate to make sacrifices.

## Screening

We have a standardised screening process that candidates undergo. It's a brief call with some basic questions. A lot of it is just getting basics out of the way:

* Preferred name
* Pronouns
* Best contact method
* What should we use for work history review (resume, LinkedIn)
* An open source project that the candidate is excited to talk about`*`

Once we have that taken care of, we'll ask some questions to get to know the candidate better. This is a candidate's opportunity to get an advocate in the room. The interviewer doing the screening call is looking for information about the candidate's background, knowledge, and expertise. The interviewer will then carry this info over to the next step of the process so they can advocate for the candidate prior to the interview.

If the screening confirms that the candidate covers the basic needs of the role they're applying for, we'll send candidates an invitation to schedule an interview. On the other hand, we'll let candidates an know if we decide not to move forward at this point. The last thing we want to do is leave a somebody hanging.

<small><code>*</code> If the candidate doesn't have any open source work to show off, they can instead select something from one of the many <a href="https://github.com/loopholelabs/">Loophole Labs OSS projects</a>. We prefer candidates send us one of their own projects because it requires less work from them, but this alternative is perfectly acceptable.</small>

## Pre-Interview Sync

We schedule this internal meeting at least 2 days before the interview. We'll use the time to assess what knowledge gaps we have around the candidate. The screening interviewer will take the info they learned during the screening call and use it to fill in some of these gaps. We'll try to fill other gaps by reviewing the candidate's work history, website, open source work, etc.

The goal here is to generate a set of questions that are specific to the candidate. Some of our standard interview questions will even be skipped if we feel we have enough positive signal to answer them ahead of the interview!

Once the meeting is over, we compile our questions into an agenda, export it to a PDF, and send it to the candidate at least 24 hours before their interview. We expose every question that we intend to ask beforehand because it gives the candidate time to think about the questions. It makes it easy for candidates to bring their full self to the interview, knowing exactly what we're going to be discussing. It also eliminates a level of stress and anxiety for most candidates because they don't have to guess what they'll be asked during an interview; it's all right there in the agenda. ❤️

## Interview

The actual interview is broken into 4 sections: Overview, Technical Discussion, Team Q&A, and CEO Q&A.

### Overview

_Expected time limit: 15 minutes_

This first part of the interview is super basic. Both the candidate and the interviewers will introduce themselves, then we'll recap the role the candidate is interviewing for. We'll also do a brief recap of the sections of the interview before we dive in.

### Technical Discussion

_Expected time limit: 30 minutes_

We'll ask questions about the submitted OSS project after the screening call. Rather than digging into technical acuity (which can be trained), we're trying to better understand the candidate's ability to communicate about technical topics. Some example questions:

* Why build `𝑥` instead of using library `𝑦`?
* What was the biggest technical hurdle?
* How was the break down and prioritisation of work managed?

The goal is to make sure that the candidate can break technical topics down in such a way that they can communicate them to us, and we can actually understand what they're conveying.

### Team Q&A

_Expected time limit: 15 minutes_

During this time, the CEO will leave the call so the candidate feels free to ask questions of the rest of the team. Candidates are encouraged to ask difficult questions, such as, "What's it like to work for the CEO," or, "How do you feel about the stability of the org?" Lending back to Principle 2, _transparency is key_. We want people to work for us, but we don't want them to show up and find that we can't meet their needs as a work place.

### CEO Q&A

_Expected time limit: 15 minutes_

The final leg of the interview has the CEO return to the call while the rest of the team heads off. Candidates can certainly ask the same questions of the CEO that they asked of the team, but they're encouraged to dive further into things like...

* What specific projects would I be working on in this role?
* How solid is the company's funding?
* What's the company's growth plan over the next year?

On the flip side, our CEO will also take this opportunity to better understand candidates' goals and needs, and assess whether or not our company can fulfil them. Again, we want to be able to set clear expectations, and we don't want somebody to accept an offer only to feel like the rug was pulled from under them when things aren't exactly what they expected.

## Post-Interview Sync

We try to get this internal meeting on the calendar right after the interview. While a lot of companies handle post-interviews with solo interviewer surveys or arbitrary discussions, we have a single document that guides us through the process of filling in the gaps that we found during the pre-interview sync. Once we're done, we'll make a hiring decision. This decision comes down to 2 questions:

1. Do we believe the candidate can accomplish the goals we have for this role?
1. Do we believe we can collaborate effectively with this candidate?

If the answer to both of these questions is "yes," we're ready to send an offer! Otherwise we'll respond and let the candidate know we're not moving forward, and we try to do that as soon as possible.

## Offer Negotiation

The heading here is a bit of a misnomer; there's actually no negotiation in our offers. That said, it's not a take–it–or–leave–it or a one–size–fits–all offer, either. What we present is a salary range alongside an equity range (both listed in the job descriptions). The candidate can then choose where where they want to land in both ranges, where the equity amount is inversely proportional to the base salary that the candidate chooses.

Let's simplify this by looking at a hypothetical example. We'll assume we have job description with a salary range of USD$100,000 to USD$150,000 and an equity range from 0.5% to 1%. If a candidate chooses the low end of the salary range (USD$100,000), they'd get the top end of the equity range. If they were to decide they want to be in the middle, they could take USD$125,000 with 0.75% equity. Again, _these numbers are purely hypothetical_. Please see our actual job descriptions for real numbers, and don't hesitate to reach out to us if you're considering applying but you're having trouble understanding the way this scales!

# Final Thoughts

We've built this process based on our own terrible experiences interviewing in the tech industry, and as a result we've come up with something we're really proud of. Of course, we're not infallible and we recognise that there are likely issues we're missing, or biases that we haven't yet addressed. We're always looking for places to improve, so if you have suggestions please don't hesitate to let us know! The best way to do so would be to [join our Discord server](https://loopholelabs.io/discord/).

If you like how everything sounds and you're looking for your next role, make sure to check out our [current openings](https://loopholelabs.io/jobs)! We're looking forward to hearing from you. 🥰

## Credit Where Credit's Due

Our process was heavily influenced by [Jonathan](https://twitter.com/jspsun) and myself ([Trezy](https://twitter.com/TrezyCodes)). Jonathan is no longer at Loophole, but we wanted to acknowledge his significant contributions during the process of designing our interview pipeline. ❤️



For those that have followed us for a while, you may or may not be aware that we previously had an open invite to join our Slack server. We've had a handful of people trickle in, but there's been no activity in the public channels. While we _wanted_ to build a community there, we weren't being _intentional_ about doing so.

Today, all of that changes. Allow me to introduce you to the new [Loophole Labs community](https://loopholelabs.io/discord). ❤️

## Taking a Community-Centred Approach

Most companies focus primarily on customers. For the most part, that makes sense! If you want to make money, you need to cater to the people that are willing to give you money.

At Loophole, we've decided to take a different approach. We want all of our products to be open source because we want _everyone_ to be able to benefit from our work, but there's no point in making open source software if it doesn't serve a _community_.

To that end, we want to be _intentional_ about building a community. We want to ensure that we're providing tools that serve _all_ of our users... not just the ones with deep pockets. From our experience, the best way to accomplish this is with [Discord](https://discord.com).

## Intentionality

We wanted to create a platform to provide our users, our followers, and our friends with a safe place to interact with us, as well as with each other. We wanted it to be a place to ask for help, to show off your projects, and to keep up with the latest things that we're working on. Most importantly, we wanted it to be a place for _us_ to interact directly with _you_.

While we could have, maybe, _possibly_ been able to achieve this with Slack, we believe our new Discord server will do an excellent job of handling all these things and more. We're still working out everything we'll be doing on our Discord, but here's a small taste of what you can expect:

* **Community Events**
  It's always great to know other people in the community! To help everyone get to know each other, we'll be hosting regular events (such as meet and greets) on Discord.
* **Regular Status Updates**
  We'll certianly be posting about big news and releases on our blog, but we'll also be keeping the community up-to-date with more frequent, byte-sized updates about what we're working on and how it's going.
* **Team AMAs**
  We're a small and growing company, but everyone on the team is *awesome*. We'll be hosting AMAs with individual members of the team so that you can get immediate answers about what we're working on, and how our team thinks about the challenges we're trying to solve!

## How Do I Join?

It's easy! You can click [this link](https://loopholelabs.io/discord) to be redirected to Discord. If you already have the app, it should open up for you! If not, no worries. Discord runs in the browser, too, so you should be able to get started as soon as you've created a Discord account. 😉

## Final Thoughts

We are _so_ excited to see you all start joining our Discord! If you have any questions or thoughts about our move (or about anything else, for that matter), make sure to pop in to the server and drop them in the General channel! See you soon! ❤️
