Last month, secure email came under DDoS attack, this is how we added DDoS protection to Proton Mail.
Dec 16, 2015 Update: After we publicly posted the data at the end of this post, the attackers swiftly retaliated with a 59 Gbps attack. We were able to successfully mitigate with help from Radware. We knew this was a risk of disclosure, but we stand by our decision.
One month ago, we had to fight off one of the largest and most sophisticated DDoS attacks(nieuw venster) to hit Europe this year. As we detailed in our previous blog post(nieuw venster), defending against DDoS attacks, especially highly sophisticated ones, is not easy. However, Proton Mail is now proven to be capable of providing DDoS resistant email hosting for our business and consumer users. In this post, we will share some of the lessons learned to help other companies that need DDoS protection. At the end of this post, we also provide some technical details about the attack against us, along with a partial data dump.
If you would like to try out the future of encrypted email, now with DDoS protection, it is possible to sign up for a free email account.
First it is worth mentioning that DDoS attacks against Proton Mail have NOT stopped. In fact, even though we were able to mitigate the attacks since November 8th, 2015, we been under attack repeatedly since then. While smaller than the initial attack, these subsequent attacks have still reached 40-50 Gbps and as of a couple days ago, we are still being hit. Other than a few brief interruptions, we have been able to keep our services online over the past month so the DDoS protection we have put in place is effective.
As the world’s largest secure email provider, Proton Mail protects the communications of many journalists and activist groups around the world. This unfortunately also makes Proton Mail a target, and we cannot rule out the possibility that the attack against Proton Mail was state sponsored. However, DDoS attacks are on the rise, hitting record levels this year(nieuw venster), and it is inevitable that more and more companies will come under attack in the future.
The attack against Proton Mail is an anomaly because of its scale and level of sophistication, and there is very little literature available about how to defend against attacks of this sort. In this post, we will cover the key considerations, the solution we employed, and finally some technical details about the attack against us. The goal of this post is to help the security community at large defend against future attacks of this magnitude.
DDoS Protection – Key Considerations
Below are the four factors that must be considered when developing a DDoS defense strategy. It is possible to consider much less when defending against a basic attack, but against a sophisticated attacker, each of the below points is crucial.
1. Understand your legitimate traffic
How you defend against attack depends a lot on what types of legitimate traffic passes to your site. Some websites only have HTTP/HTTPS traffic which flows on port 80/443. However, a sophisticated attacker will in fact attack all aspects of your infrastructure, such as email traffic on port 25. For most companies, having email working is just as essential as having the website working. In Proton Mail’s case, this was critical because as a secure email service, we also needed to protect SMTP traffic on port 25. The need to support SMTP traffic meant that we could not use most DNS redirection DDoS solutions (such as Cloudflare).
2. You are vulnerable if your datacenter and upstream providers are vulnerable
When facing a truly large scale DDoS, the attackers WILL also go after your upstream providers. In Proton Mail’s case, all of our upstream ISPs were attacked, and in fact the entire datacenter we are located at was taken offline. In reality, it is very difficult to also ensure that your datacenter and upstream providers are able to resist a large scale attack, so the only real solution is to build your infrastructure in a way that makes you as independent as possible. Having the best protection in the world for your own site is no good if the internet connection coming to your server is no longer working because of an attack upstream.
3. Collateral damage matters
By their nature, massive attacks almost always cause collateral damage. In Proton Mail’s case, companies as far away as Moscow were knocked offline. This is bad news for several reasons. First, other impacted companies will quickly start applying pressure and worsen a bad situation.
Secondly, when given the choice between keeping you online (and risking hundreds of other companies), versus taking you offline, most service providers will opt to take your site offline. In fact, at the height of the attack against us, even if we wanted to relocate to a different datacenter, it was very unlikely that we would have found one willing to take us in given the risk of massive collateral damage following us. As with the previous point, the best defense is isolating your infrastructure.
4. Don’t compromise security/privacy
When under massive DDoS attack, sometimes security and privacy become lower priority issues, but de-prioritizing security and privacy can be a mistake. A DDoS can take your site offline, but it cannot breach its security or privacy. However, improperly implemented DDoS protection CAN breach privacy. For example, some DDoS solutions require handing over SSL keys to allow for the scanning and subsequent filtering of encrypted traffic. While this may lead to mitigating the attack, it also means the DDoS protection provider can now scan all traffic to your site, thus breaching user privacy. Due to our focus on secure email, Proton Mail had to adopt a solution that did NOT weaken email privacy.
DDoS Protection for Secure Email
The above considerations unfortunately meant Proton Mail needed to implement a more advanced solution which could not be done overnight. As a result, we suffered intermittent downtime over the period of several days while we implemented our solution. We implemented a two part solution which has been effective.
1. Isolate infrastructure
The first step was to make Proton Mail as independent as possible from our datacenter and upstream providers. This was done by getting a direct connection with a Tier 1 carrier, in our case, Level 3 Communications. Unfortunately, due to the remote location of our datacenter, this required running a direct connection 114 kilometers from the main Swiss point of presence in Zurich to our underground facility. We also needed to put in our own routers at the datacenter so that we cannot be taken offline if the datacenter edge routers come under attack. These routers were the critical equipment that we transported from Geneva to the datacenter(nieuw venster) on the day we mitigated the attack. After this was done, we effectively became our own ISP and only relied on the datacenter for power and cooling.
2. Use a cloud based DDoS mitigator
Now that we had made Proton Mail our own ISP, attacks against the datacenter or upstream ISPs could no longer impact us. However, a large DDoS can still manage to saturate our link or overload our load balancer. We needed a way to block DDoS attack traffic before it even hits our network. To do this, we went with a cloud based DDoS mitigator.
A cloud based DDoS mitigator works as follows. First, we use BGP redirection to route traffic through our DDoS mitigator, which means that among the dozen or so hops that internet traffic takes to reach our servers, one of them is the DDoS mitigator. A pre-requisite for this is that your company must be a LIR (Local Internet Registry) and have at least a /24 of PA (Provider Aggregatable) IP addresses. Fortunately, we had the foresight to join RIPE in 2014(nieuw venster) and were already a LIR so this requirement was met.
The dirty traffic is sent to the DDoS mitigator which typically has massive network capacity. The DDoS mitigator will absorb the attack traffic and send it to scrubbing centers which will filter the traffic and only pass on the clean traffic. This is then sent back to our network via redundant GRE tunnels. The data that we return to legitimate users however goes directly to the users.
(nieuw venster)After much analysis, we partnered with Radware(nieuw venster), one of the world’s top DDoS protection companies. The decision was driven by two of the considerations listed above. First, Radware’s technology can work with all of the legitimate traffic that we expect, including SMTP. Secondly, Radware’s technology does not require our SSL keys to operate effectively, meaning both layers of encryption that Proton Mail offers (SSL and OpenPGP) can be kept intact. Thus, there is no compromise in the privacy of our secure email service.
Conclusion
The attack faced by Proton Mail was highly sophisticated and unfortunately required extraordinary effort to defeat. In the next section, some technical details of the attack against us are discussed.
In defeating this attack, we were able to benefit from strong in-house technical expertise, along with a partnership with IP-Max(nieuw venster), the leading networking experts in Switzerland. Defending against large scale DDoS attacks remains an expensive undertaking. Below are the typical costs for this type of DDoS protection:
Networking equipment: $30’000
BGP/GRE DDoS Mitigation (per year): $50’000 – $100’000
Dedicated IP Transit (per year): $20’000
Maintenance Overhead: $10’000+
A lot of this is just the added complexity of running our own network with dedicated hardware, which then has to be monitored and maintained. Our overall costs came in lower than the typical costs listed above, but they were still substantial. Due to the strong financial support(nieuw venster) we received from the Proton Mail community, we were able to manage the costs. Despite the attack against us, we remain committed to protecting online privacy rights for all, and with your support, we will continue to provide secure email services to the world for free.
Proton Mail DDoS Attack Details
After much deliberation, we have decided to make some data from the Proton Mail attack available to the public. We know that disclosures such as these will often invite further attacks, but it is in the interests of the security community at large to make this data available (as such data is rarely released). Over the course of the past month, we have been hit with a wide range of attacks. The following is a partial listing of attack vectors which were utilized. A typical attack would usually utilize 5 or 6 of these attack vectors simultaneously.
- NTP-reflection
- UDP Flood – Reflective DNS
- UDP Flood – SSDP
- TCP RST Flood
- TCP SYN Flood
- TCP Out-of-State Flood
- ICMP Flood/RST Flood
- SYN Flood – HTTPS
- SYN Flood – HTTP
- SYN-ACK Flood
- UDP-fragmented
Also included below is a data dump from one of the attacks against our infrastructure.
Proton Mail DDoS Attack Data Dump(nieuw venster)
Even though they were not the main attacker in our case, we are also coordinating an international effort to track down the Armada Collective(nieuw venster), the first of the two groups that hit Proton Mail. Victims can contact us at security@proton.me
(nieuw venster)
