As the world’s largest encrypted email provider, working with IMAP (Internet Message Access Protocol) is something that we do on a daily basis. IMAP is core to the Proton Mail Bridge app, which lets you add Proton Mail encryption to standard email clients, such as Outlook, Thunderbird, or Apple Mail.
But IMAP goes beyond Proton Mail. It is one of two standard email retrieval protocols (the other being POP) that nearly every email app uses to access and manage your emails. So, in essence, it powers email globally.
Today, we’re introducing Gluon(nieuw venster), a new IMAP library written in the Go programming language designed to be high-performance, reliable, developer-friendly, and, most importantly, open-source.
Together with the launch of Gluon, we are also releasing a new version of the Proton Mail Bridge that is powered by Gluon. As a result of the innovations below, the new Proton Mail Bridge is 1000% faster, far more reliable, and also compatible with more email clients.
Why create a new IMAP library?
Email needs to be reliable, but it also needs to be high-performance, particularly as the typical inbox size has grown significantly over the past decade. Many open-source IMAP implementations tend to optimize for one and not the other, leading to pretty significant trade-offs or bugs.
Gluon seeks to bridge that gap and overcome the limitations in existing open-source IMAP libraries, which are often poorly maintained, or not entirely scalable. Gluon does this by utilizing an architecture that relies upon a “snapshot” system.
IMAP clients typically refer to messages by their “sequence number”, the message’s position in a mailbox. The first message has the sequence number “1”, the second “2”, and so on. If a client wants to mark a message as “read”, it will send a command to the server such as “mark message 5 as read”. But what if another client deleted the fourth message in the mailbox? The sequence numbers of all messages after the deleted message will be shifted down by one; the client that sent the “mark message 5 as read” command now refers to a different message than it intended.
IMAP servers (which include applications such as the Proton Mail Bridge) need to be able to handle this situation. When one client moves messages into or out of a mailbox, the server needs to notify all other clients of the changes so that they can update their own view of the mailbox. And until the clients have received the update, the server needs to remember what each client thinks the mailbox looks like to correctly interpret the client’s commands.
In the previous example, the server needs to know that the client that sent the “mark message 5 as read” command, is referring to the message that was originally in position 5, not the message that is currently in position 5.
This type of scenario can occur more frequently in modern email usage, where the user might be using Proton Mail on web on one device, using the mobile apps on the go, and then using a desktop client via Proton Mail Bridge on a desktop, all of which may not be online at the same time.
Another scenario is email apps that often utilize multiple simultaneous connections to your mailbox to speed things up, but this can then result in concurrency issues. By using a snapshot system, Gluon assigns each IMAP client its own “snapshot” of the selected mailbox. Each snapshot holds the client’s unique view of the mailbox, allowing the server to interpret exactly which message the client is referring to at any point in time, regardless of what actions have been performed by other clients. This guarantees a stable and consistent email experience for the user.
How we wrote Gluon
Our first step in writing Gluon was to generate an IMAP parser from the syntax given in RFC3501(nieuw venster). We used ANTLR4(nieuw venster), a popular parser generator, to create a parser that could parse IMAP commands and responses according to the specification. This allowed us to focus on implementing the logic of the IMAP protocol rather than parsing and validating input.
Once we had a parser, we wrote the basic server type for Gluon. The server type waits for incoming TCP connections and spawns an “IMAP session” running in a separate goroutine (a lightweight, green thread used in the Go programming language) to handle each connection.
The session has a simple job:
- Read a client’s command.
- Parse the command.
- Call the right command handler.
- Finally, send any necessary response(s) to the client.
This design also allows Gluon to handle multiple client connections concurrently, with each session managing its own state.
One of the key challenges in implementing an IMAP server is managing both the persistent states and per-session states of mailboxes. The persistent state refers to the messages that are actually in a selected mailbox while the per-session state refers to the messages each client thinks are currently in a selected mailbox.
In Gluon, we use an SQL database to store the persistent IMAP state, such as which mailboxes and messages a user has. Furthermore, the SQL database allows for faster and more efficient handling of commands thanks to intelligent pre-fetching and indexing.
Managing the per-session state was more complicated, as it depends entirely on which IMAP responses have been sent to a client at a given point in time. To model this, we defined a type that holds a list of message IDs, UIDs, and flags in memory. This list is populated from the database when a client first selects a mailbox. This approach allows us to efficiently manage the per-session state and handle many IMAP commands entirely in memory without requiring disk reads, leading to much faster performance.
To synchronize the per-session state between multiple connected clients, Gluon uses a system of “responders”. These are types that encapsulate a change of state and, when executed, are converted into IMAP responses. When a client performs an action (such as marking a message as read) that would change the state of another client, the backend creates a responder for the action and pushes it to the affected state. The affected state remains unchanged until the responder is executed, at which point it is updated, and a corresponding IMAP response is sent to the client. This approach allows Gluon to efficiently manage the per-session state while ensuring consistency across multiple clients.
While building up Gluon’s support for each IMAP command, we used test-driven development. We created a testing framework that allowed us to specify what an entire IMAP session should look like, specifying client commands and expected server responses.
We first wrote a test for each IMAP command (often copied directly from RFC3501) and then implemented the command handling to make the test pass. Furthermore, to ensure Gluon’s correctness and reliability, we used Dovecot(nieuw venster), the world’s most popular IMAP server, as a reference implementation, and we performed correctness testing using Dovecot’s own testing tool, imaptest(nieuw venster).
The last step was to integrate Gluon into the Proton Mail Bridge. We designed Gluon so that integrating it into any application would be as simple as implementing its “Connector” interface. The Connector keeps the Gluon and external states (the Proton state) in sync.
For example, when an IMAP client marks a message as read, the connector marks that same message as read on the Proton server. When the Proton server receives a message, the connector downloads and decrypts that message and puts it in the right Gluon mailbox.
This extensible design makes it possible to use Gluon with almost any application that requires IMAP.
Privacy without compromising performance
Earlier this year, we put the new version 3 of Proton Mail Bridge (powered by Gluon) into beta testing, and the user feedback was in line with our own performance testing which indicates a 1000% speed improvement. We hope that by releasing Gluon as open-source software, we can enable a new generation of modern email software better able to handle the demands of contemporary email users.
As an open-source company, we welcome others to use, review and contribute to the code, and as with other other open-source projects that Proton maintains(nieuw venster), we are committed to maintaining this library over the long term.
Our mission is to make privacy accessible and widely available online, and a better Proton Mail Bridge powered by Gluon that makes end-to-end encrypted email available on any desktop email application is an important step towards achieving that goal.
As always, thank you for your support, and don’t forget to share your feedback and thoughts with us on Reddit(nieuw venster), and Twitter(nieuw venster).
This work was conducted by James Houlahan, Leander Beernaert, Jakub Cúth, Xavier Michelon, Romain Le Jeune, Gjorgji Slamkov, Alexander Khusanov, Gabor Meszaros and Andrzej Szafranski from the Proton Mail team.
