(Version française ci-dessous)
EDIT: We have given an exclusive interview to Tech Republic about the DDoS attacks and what happened behind the scenes. The story is now online here: http://www.techrepublic.com/article/exclusive-inside-the-protonmail-siege-how-two-small-companies-fought-off-one-of-europes-largest-ddos/(new window)
We are happy to announce today that after several days of intense work, we have largely mitigated the DDoS attacks against us. These attacks took Proton Mail offline making it impossible to access emails, but did not breach our security. At present, attacks are continuing, but they are no longer capable of knocking Proton Mail offline for extended periods of time. As our infrastructure recovers over the next several days, there may still be intermittent service interruptions, but we have now largely restored all services. Our successful recovery was only possible due to the valiant efforts of IP-Max(new window) and Radware(new window), and we would like to sincerely thank them.
It has now been one week since the first attack was launched against Proton Mail. Since then, we have been subject to the largest and most extensive cyberattack in Switzerland, with hundreds of other companies also hit as collateral damage. In addition to hitting Proton Mail, the attackers also took down the datacenter housing our servers and attacked several upstream ISPs, causing serious damage. More about the attack which hit us can be found in our earlier statement which is copied below.
Throughout the past week, our team has worked tirelessly to restore service because we know in addition to over half a million regular users, there are also activists, dissidents, and journalists who rely upon Proton Mail for their communications. Mitigating an attack of this size and complexity is not easy and while we managed to restore services several times throughout the week, it was not until around 3 AM Geneva time on November 8th that we finally managed to gain control over the situation.
Rescuing Proton Mail was not a solo effort, and we will forever be grateful to everyone who came to our assistance. Within Switzerland alone, dozens of companies and individuals came forward to help us once news of the attack spread. In particular, we want to thank the network experts from IP-Max in Geneva who volunteered their time and expertise. Without their heroic 18 hour effort, it would not have been possible for Proton Mail to come back up so quickly in the face of such a massive attack. During the rescue operation, the IP-Max team accomplished the impossible, and managed to connect a brand new direct line from our datacenter to the main PoP in Zurich, 114 kilometers away, on a Saturday, in less than 18 hours! Nobody else in Switzerland could have accomplished that. We would also like to thank Level 3 Communications for arranging an emergency IP Transit, and Patrick Muller, Pim van Pelt, David Corriveau, Ivan Adji-Krstev, and Michel Streiff for their assistance.
In order to mitigate the DDoS attack against us, we partnered with Radware, one of the world’s premier DDoS protection companies. In Radware, we found a solution that was capable of protecting Proton Mail without compromising email privacy. Given the magnitude of the attack we faced, we knew that we would have to work with the best, and Radware’s BGP redirection solution fit our requirements. During our hour of need, there were many companies who attempted to charge us exorbitant amounts, but Radware offered their services at a very reasonable price in order to get us online as soon as possible. With Radware DefensePipe, we were finally able to mitigate the attack on Proton Mail.
Last but not least, we would like to thank our community of users. We built Proton Mail for you, and we would like to thank you for having our back, in both good times and bad times. In just three days, the Proton Mail Defense Fund has gathered $50,000(new window) in donations, giving us the resources to resist further attacks against email privacy. By attacking the world’s largest free private email service, the attackers sent a message that they did not want online privacy to succeed. However, we have now sent them back an even stronger message, that online privacy is here to stay.
Today, Proton Mail is stronger than ever, and with IP-Max(new window) and Radware(new window) behind us, not only did we mitigate the largest DDoS attack in Switzerland in a couple days, we also gained the ability to resist such attacks in the future. The attackers hoped to destroy our community, but this attack has only served to bring us all together, united by a common cause and vision for the future. The road ahead will surely contain more difficulties, but together, we shall overcome.
What’s Next?
In the coming weeks, we will be analyzing the data from the attack against us with the assistance of experts from around the world and releasing a report. Here is what we can safely disclose at this time:
-
- Proton Mail was attacked by at least two separate groups. The first attacker, the Armada Collective, demanded a ransom, more on this can be found in the previous posts copied below. The Armada Collective has contacted us to deny responsibility for the second attack.
-
- The second group caused the vast majority of the damage, including the downing of the datacenter and crippling of upstream ISPs, exhibiting capabilities more commonly possessed by state-sponsored actors. They never contacted us or made any ransom demands. Their sole objective was to take Proton Mail offline, at any cost, with no regards for collateral damage, and to keep us offline for as long as possible. They have still not been identified.
-
- It was not until the 3rd day of attack that we realised there were two separate attackers. Given the sophistication of the attack used by the second group, we believe they may have been preparing their attack against us for some time. After seeing the first attack, they chose to strike immediately afterwards in the hopes that they would not be discovered as being a separate attacker.
-
- If there are DDoS experts interested in reviewing the attack data, we welcome them to contact us at security@proton.me
At some point in the future, we will also share the full story about the heroics of November 7th.
Unfortunately, this attack will set back our development timeline so releasing Proton Mail 3.0 at the end of November will no longer be possible. Once we have reworked our development timeline, we will advise what the new release date will be. We look forward to continuing on the journey towards a more private and free internet with all of you.
Copied below, you will find our original statements from November 5th.
Proton Mail Statement about the DDOS Attack(new window)
As many of you know, Proton Mail came under sustained DDOS attack starting on November 3rd, 2015. At the current moment, we are not under attack and have been able to restore services, but we may come under attack again.
We are currently working with solution providers to find a way to mitigate this attack, however, it is quite unprecedented in size and scope so unfortunately finding a working solution is not easy. Because of the sophistication of this attack, we will also need to resort to quite expensive solutions which will burden our finances. It is for this reason that we are also collecting donations for a Proton Mail defense fund.
Donate to the Proton Mail Defense Fund(new window)
Proton Mail was originally created to provide privacy to activists, journalists, whistleblowers, and other at risk groups, and we have many of those people in the Proton Mail community. Unfortunately, there are groups out there determined to oppose this which has led to this incident. However, we are confident that with your support, we can overcome this attack and come back stronger than ever, and continue to provide a place where online privacy is protected.
As we will detail below, this attack has grown beyond just Proton Mail and is a full fledged cyberattack. We have been working with the Swiss Governmental Computer Emergency Response Team (GovCERT), the Cybercrime Coordination Unit Switzerland (CYCO), as part of an ongoing criminal investigation being conducted here in Switzerland and with the assistance of Europol. After much consultation, we have decided to release details about the full extent of the attack on us so the broader security and privacy community can stay informed.
Slightly before midnight on November 3rd, 2015, we received a blackmail email from a group of criminals who have been responsible for a string of DDOS attacks which have happened across Switzerland in the past few weeks(new window).
This threat was followed by a DDOS attack which took us offline for approximately 15 minutes. We did not receive the next attack until approximately 11AM the next morning. At this point, our datacenter and their upstream provider began to take steps to mitigate the attack. However, within the span of a few hours, the attacks began to take on an unprecedented level of sophistication.
At around 2PM, the attackers began directly attacking the infrastructure of our upstream providers and the datacenter itself. The coordinated assault on our ISP exceeded 100Gbps and attacked not only the datacenter, but also routers in Zurich, Frankfurt, and other locations where our ISP has nodes. This coordinated assault on key infrastructure eventually managed to bring down both the datacenter and the ISP, which impacted hundreds of other companies, not just Proton Mail.
At this point, we were placed under a lot of pressure by third parties to just pay the ransom, which we grudgingly agreed to do at 3:30PM Geneva time to the bitcoin address 1FxHcZzW3z9NRSUnQ9Pcp58ddYaSuN1T2y. This was a collective decision taken by all impacted companies, and while we disagree with it, we nevertheless respected it taking into the consideration the hundreds of thousands of Swiss Francs in damages suffered by other companies caught up in the attack against us. We hoped that by paying, we could spare the other companies impacted by the attack against us, but the attacks continued nevertheless. This was clearly a wrong decision so let us be clear to all future attackers – Proton Mail will NEVER pay another ransom.
Through MELANI (a division of the Swiss federal government), we exchanged information with other companies who have also been attacked and made a few discoveries. First, the attack against Proton Mail can be divided into two stages. The first stage is the volumetric attack which was targeting just our IP addresses. The second stage is the more complex attack which targeted weak points in the infrastructure of our ISPs. This second phase has not been observed in any other recent attacks on Swiss companies and was technically much more sophisticated. This means that Proton Mail is likely under attack by two separate groups, with the second attackers exhibiting capabilities more commonly possessed by state-sponsored actors. It also shows that the second attackers were not afraid of causing massive collateral damage in order to get at us.
At present, Proton Mail’s infrastructure is still vulnerable to attacks of this magnitude, but we have a comprehensive long term solution which is already being implemented. Protecting against a highly sophisticated attack like the second one which was launched against us requires sophisticated solutions as we also need to protect our datacenter and upstream providers. Cost estimates for these solutions are around $100,000 per year since there are few service providers able to fight off an attack of this size and sophistication. These solutions are expensive and take time to implement, but they will be necessary because it is clear that online privacy has powerful opponents. In order to cover these costs, we are collecting donations for a Proton Mail defense fund, which can be found here:
Donate to the Proton Mail Defense Fund(new window)
We are fighting not just for privacy, but for the future of the internet. We would especially like to thank the thousands of users who offered their support and encouragement on Twitter and Facebook, we will never stop fighting for you. Over the next several weeks, we will begin putting in place the sophisticated protections that are necessary to withstand large scale attacks like this to ensure that online privacy can’t be taken down.
DDOS Update
We’re sorry that we were unable to prevent this from happening and we are determined to get everyone access to their email as soon as possible.
On Tuesday November 3, 2015 Proton Mail was taken offline by an extremely powerful DDOS attack.
For people who don’t know what a DDOS attack is, here is a metaphor that best illustrates it:
Imagine yourself as a car on the freeway. You want to access Proton Mail, so you are driving to visit our site that’s located in Switzerland. Because the internet is amazing, it takes less than a second to arrive. During a DDOS attack, millions of fake cars join you on the freeway and cause a massive traffic jam. The result is that Proton Mail is unharmed and perfectly fine, but no one can visit because of the grid-lock.
The attackers began by flooding our IP addresses. That quickly expanded to the datacenter in Switzerland where we have our servers. In the process of attacking us, several other tech companies and even some banks were knocked offline temporarily.
Despite our best efforts, we have been unable to stop the attack but we are working non-stop to get back online.
Even though access is limited, an important thing to note is that our core end-to-end encryption holds strong and is 100% untouched. All user data is fine and safe.
To solve this problem we are working with the top companies and people both onsite in our Swiss data center and from around the world. We are confident we will be back online – we just wish it was sooner rather than later.
For the latest updates, Twitter is the best place to look.
Version Française
Nous sommes heureux d’annoncer aujourd’hui qu’après plusieurs jours de travail intense, nous avons largement mitigé les attaques DDoS qui nous frappaient. Ces attaques ont rendu Proton Mail inaccessible mais n’ont pas mis en danger la sécurité de nos services. A présent, les attaques continuent, mais ne sont plus en mesure de rendre Proton Mail inaccessible pour de longs moments. Alors que nous adaptons notre infrastructure au cours des prochains jours, il se peut que quelques problèmes d’accès se produisent, mais nous avons maintenant largement restauré tous les services. La réussite de ce rétablissement de nos services n’a été possible que grâce aux efforts vigoureux d’IP-Max et de Radware, que nous tenons à remercier sincèrement.
Cela fait maintenant une semaine que la première attaque a été lancée contre Protonmail. Depuis, nous avons subi la plus grande et la plus longue cyber-attaque de Suisse, affectant par dommages collatéraux des centaines d’autres entreprises. En plus de l’attaque contre Protonmail, les attaquants ont également rendu inaccessible le centre de données hébergeant nos serveurs, et attaqué plusieurs de nos fournisseurs d’accès Internet, produisant des dommages considérables.
Pendant cette dernière semaine, notre équipe a travaillé sans relâche pour rétablir le service, car nous savons que, parmi notre demi million d’utilisateurs, il y a des activistes, des dissidents et des journalistes qui dépendent de Protonmail pour communiquer. Mitiger une attaque de cette ampleur et de cette complexité n’est pas chose facile, et alors que nous avions réussi plusieurs fois à rétablir le service pendant la semaine, ce n’est qu’à 3 heures du matin le 8 novembre, heure de Genève, que nous avons finalement repris le contrôle de la situation.
Sauver Protonmail n’a pas été un effort solitaire, et nous serons toujours reconnaissant envers tous ceux qui nous ont aidé. Rien qu’en Suisse, des douzaines d’entreprises et de particuliers nous ont offert leur aide, dès que l’information concernant l’attaque s’est propagée. En particulier, nous voulons remercier les experts réseau d’IP-Max à Genève, qui ont donné leur temps et leur expertise. Sans leurs efforts héroïques pendant 18 heures, il n’aurait pas été possible pour Proton Mail de revenir en ligne aussi rapidement, en face d’une attaque aussi massive. Pendant l’opération de sauvetage, l’équipe d’IP-Max a accompli l’impossible, et a réussi à connecter une nouvelle liaison entre le datacenter et un gros point de présence à Zürich, 114km plus loin; un samedi, en moins de 18 heures. Personne d’autre en Suisse n’aurait pu accomplir ceci. Nous aimerions également remercier Level 3 Communications pour avoir arrangé un transit IP en urgence, et également Patrick Muller, Pim van Pelt, David Corriveau, Ivan Adji-Krstev et Michel Streiff pour leur précieuse aide.
Afin de mitiger les attaques DDoS qui nous frappaient, nous avons établi un partenariat avec Radware, une des premières entreprises de protection contre les dénis de service au monde. Avec Radware, nous avons trouvé une solution capable de protéger Protonmail, sans compromettre la confidentialité des emails. Vu l’ampleur de l’attaque que nous avons affrontée, nous savions que nous devions faire appel au meilleur, et la solution Radware de redirection BGP répond à notre attente. Au plus fort de notre besoin, plusieurs entreprises ont essayé de nous soutirer des montants exorbitants, mais Radware a offert ses services à un prix très raisonnable, afin que nous soyons de retour en ligne aussi vite que possible. Avec Radware DefensePipe, nous avons finalement pu mitiger l’attaque sur Protonmail.
Finalement, nous voulons remercier notre communauté d’utilisateurs. Nous avons construit Protonmail pour vous, et nous souhaitons vous remercier pour votre soutien, autant dans les bons que dans les mauvais moments. En à peine trois jours, le fond de défense de Protonmail a réuni 50’000$ de donations, nous offrant les ressources pour résister à de prochaines attaques contre le droit à la confidentialité des emails. En ciblant le plus grand service gratuit d’email sécurisé, les attaquants ont montré leur volonté d’anéantir la confidentialité en ligne. Nous leur avons maintenant envoyé une réponse encore plus forte, à savoir que la confidentialité en ligne est là pour durer.
Aujourd’hui, Proton Mail est plus fort que jamais, et avec IP-Max et Radware qui nous soutiennent, nous avons non seulement déjoué la plus grande attaque en Suisse en quelques jours, mais nous avons aussi acquis la capacité à résister à de telles attaques dans le futur. Les assaillants ont espéré détruire notre communauté, mais cette attaque a surtout permis de tous nous rassembler, unis par une cause et une vision du futur partagées. Le chemin à parcourir sera sûrement encore parsemé d’embûches, mais ensemble, nous vaincrons.
Sign up and get a free encrypted email account from Proton Mail.
