Downtime for August 11, 2014

Jason Stockman

Share this page

UPDATE August 12, 2014We have managed to recover most of the missing emails, and will work to restore them over the next couple days.

On Monday, August 11, 2014 we took Proton Mail offline for upgrades, which we estimated would take just a few minutes. Part of the upgrades for Monday were to optimize our database table structure, which would speed up the performance of the site for all users.

Unfortunately, a wrong command typed by one of our developers resulted in corrupting the messages table which resulted in losing messages for most of Sunday and the earlier part of Monday (about 20 hours). Needless to say this was a terrible mistake and we’re very sorry that this happened.

This seemingly benign update took a terrible turn because our standard operating procedure (SOP) wasn’t followed. Because a backup was not made prior to beginning database operations (as mandated by our SOP), we lost a day’s worth of user data. The Proton Mail team is growing right now, and unfortunately, we did not properly communicate our standard operating procedures to some of the new team members.

In the next few weeks we will be rolling out a more robust server architecture and deployment process that will not only speed up the site, but also increase reliability, reduce the need for downtime and prevent these sort of errors from happening. We will also be increasing oversight of staff and centralizing operations to improve our internal communications and ensure our established procedures are always followed.

These procedures have helped Proton Mail avoid data loss and extended downtime up until today, and could have prevented the incident today.

The road to bringing privacy back to email is a challenging one, but its all been possible thanks to your support. Over the coming weeks, we will be building in multiple safeguards to prevent data loss in the future. Thanks for bearing with us through our beta and understanding that human errors like this do happen.

Protect your privacy with Proton
Get a free account

Share this page

Jason Stockman

Jason is a co-founder of Proton Mail. He works on building Proton Mail's user interface and websites. Jason has over 10 years experience building websites and applications.

Related articles

The first month of 2023 has brought brutal layoffs from Big Tech, a potential ban of TikTok in the US, and another Twitter breach. But the biggest development of this new year has to be the ascent of ChatGPT.  The chatbot can produce remarkably huma
Hackers were able to steal account details from over 200 million Twitter users and posted the database on a hacking forum in early January 2023. These details include users’ email addresses and Twitter handles, allowing people to potentially identify
From your online shopping receipts to financial statements, your emails contain a great deal of sensitive information about your life, interests, and daily schedule. If you’re concerned about your online privacy, it’s therefore vital to keep your inb
At Proton, we’re committed to building privacy-focused products that are convenient to use and improve your productivity. Last year, we released the new mobile apps for Proton Calendar and Proton Drive, letting you manage your schedule and upload imp
Most email services aren’t secure and limit attachment file sizes, but there are ways to send large files securely. If you’ve ever tried attaching multiple images or video files to an email, you’ll know that it doesn’t always work. We explain ways t
Email wasn’t initially designed to be secure. From spam and phishing attempts to malware, unethical marketers and cybercriminals try to undermine the security and privacy of your inbox every day. Since your inbox stores plenty of sensitive informatio