all-in-one privacy solution":["Proton Unlimited jest kompleksowym rozwiązaniem zapewniającym prywatność"],"Black Friday":["Czarny Piątek"],"No ads. Privacy by default.":["Brak reklam. Prywatność w standardzie."],"People before profits":["Ludzie przed zyskiem"],"Security through transparency":["Bezpieczeństwo dzięki przejrzystości"],"The best Proton Mail ${ BLACK_FRIDAY } deals":["Najlepsze oferty Proton Mail ${ BLACK_FRIDAY }"],"The world’s only community- supported email service":["Jedyna na świecie usługa e-mail wspierana przez społeczność"]},"specialoffer:limited":{"${ hours } hour":["${ hours } godzina","${ hours } godziny","${ hours } godzin","${ hours } godziny"],"${ hoursLeft }, ${ minutesLeft } and ${ secondsLeft } left":["Pozostało: ${ hoursLeft }, ${ minutesLeft } i ${ secondsLeft }"],"${ minutes } minute":["${ minutes } minuta","${ minutes } minuty","${ minutes } minut","${ minutes } minuty"],"${ seconds } second":["${ seconds } sekunda","${ seconds } sekundy","${ seconds } sekund","${ seconds } sekundy"],"Limited time offer":["Oferta ograniczona czasowo"]},"specialoffer:listitem":{"Create multiple addresses":["Utwórz wiele adresów"],"Hide-my-email aliases":["Aliasy hide-my-email"],"Quickly unsubscribe from newsletters":["Szybko anuluj subskrypcję biuletynów"],"Use your own domain name":["Korzystaj z własnej nazwy domeny"]},"specialoffer:logos":{"As featured in":["Opinie naszych klientów"]},"specialoffer:metadescription":{"Get an encrypted email that protects your privacy":["Uzyskaj zaszyfrowaną usługę e-mail, która chroni Twoją prywatność"]},"specialoffer:metatitle":{"Proton Mail Black Friday Sale - Up to 40% off":["Wyprzedaż Proton Mail z okazji Czarnego Piątku – uzyskaj do 40% zniżki"]},"specialoffer:newmetadescription":{"Get up to 40% off Proton Mail subscriptions this Black Friday. Find great deals on our secure end-to-end encrypted email plans.":["Uzyskaj do 40% zniżki na subskrypcje Proton Mail w ten Czarny Piątek. Znajdź wspaniałe oferty naszych bezpiecznych planów usługi poczty elektronicznej w szyfrowaniu end-to-end."]},"specialoffer:newmetatitle":{"Proton Mail Black Friday sale | Up to 40% off secure email":["Wyprzedaż Proton Mail z okazji Czarnego Piątku | Do 40% zniżki na bezpieczną pocztę elektroniczną"]},"specialoffer:note":{"* Billed at ${ TOTAL_SUM } for the first year":["* Płatne ${ TOTAL_SUM } za pierwszy rok"],"*Billed at ${ TOTAL_SUM } for the first 2 years":["* Płatne ${ TOTAL_SUM } za pierwsze 2 lata"],"30-day money-back guarantee":["30-dniowa gwarancja zwrotu pieniędzy"],"Billed at ${ TOTAL_SUM } for the first 2 years":["Płatne ${ TOTAL_SUM } za pierwsze 2 lata"],"Billed at ${ TOTAL_SUM } for the first year":["Płatne ${ TOTAL_SUM } za pierwszy rok"],"You save ${ SAVE_SUM }":["Oszczędzasz ${ SAVE_SUM }"]},"specialoffer:off":{"${ PERCENT_OFF } off":["-${ PERCENT_OFF }"]},"specialoffer:testimonial":{"I love my ProtonMail":["Uwielbiam ProtonMail"],"My favorite email service":["Moja ulubiona usługa e-mail"],"Thanks Proton for keeping us all safe in the complicated internet universe.":["Dziękuję Protonowi za dbanie o nasze bezpieczeństwo w skomplikowanym internetowym uniwersum."],"You get what you pay for. In the case of big tech, if you pay nothing, you get used. I quit using Gmail and switched to @ProtonMail":["Dostajesz to, za co płacisz. W przypadku przedsiębiorstw big tech, jeśli nie płacisz nic, zostaniesz wykorzystany. Przestałam korzystać z Gmail i przeniosłam się na @ProtonMail"]},"specialoffer:time":{"Days":["Dni"],"Hours":["Godz."],"Min":["Min"]},"specialoffer:title":{"And much more":["I wiele więcej"],"Safe from trackers":["Żadnych elementów śledzących"],"Stay organized":["Dbaj o dobrą organizację"],"Black Friday email deals":["Oferty e-mailowe na Czarny Piątek"],"Don’t just take our word for it":["Nie musisz wierzyć nam na słowo"],"Make your inbox yours":["Niech skrzynka odbiorcza należy tylko do Ciebie"],"Our story":["Nasza historia"],"Transfer your data from Google in one click":["Przenieś dane z Google jednym kliknięciem"]},"specialoffer:tooltip":{"Access blocked content and browse privately. Includes ${ TOTAL_VPN_SERVERS }+ servers in ${ TOTAL_VPN_COUNTRIES }+ countries, connect up to 10 devices, access worldwide streaming services, malware and ad-blocker, and more.":["Uzyskaj dostęp do zablokowanych treści i przeglądaj prywatnie Internet. Obejmuje ponad ${ TOTAL_VPN_SERVERS } serwerów w ponad ${ TOTAL_VPN_COUNTRIES } krajach z możliwością połączenia do 10 urządzeń, a także dostęp do globalnych usług streamingu, blokadę reklam, złośliwego oprogramowania i wiele więcej."],"Easily share your calendar with your family, friends or colleagues, and view external calendars.":["Łatwo udostępniaj swój kalendarz rodzinie, znajomym oraz współpracownikom i wyświetla kalendarze zewnętrzne."],"Includes support for 1 custom email domain, 10 email addresses, 10 hide-my-email aliases, calendar sharing, and more.":["Zawiera obsługę 1 niestandardowej domeny e-mail, 10 adresów e-mail, 10 aliasów hide-my-email, udostępnianie kalendarza i nie tylko."],"Includes support for 3 custom email domains, 15 email addresses, unlimited hide-my-email aliases, calendar sharing, and more.":["Obejmuje obsługę 3 niestandardowych domen e-mail, 15 adresów e-mail, nielimitowane aliasy hide-my-email, udostępnianie kalendarza i nie tylko."],"Manage up to 25 calendars, mobile apps, secured with end-to-end encryption, 1-click calendar import from Google, and more.":["Zarządzanie nawet 25 kalendarzami, aplikacje mobilne, bezpieczne szyfrowanie metodą end-to-end, importowanie kalendarza za pomocą jednego kliknięcia z Google i więcej."]},"Status banner":{"Learn more":["Dowiedz się więcej"],"Please note that at the moment we are experiencing issues with the ${ issues[0] } service.":["Uwaga: obecnie występują problemy z usługą ${ issues[0] }."],"We are experiencing issues with one or more services at the moment.":["Mamy problemy z co najmniej jedną usługą."]},"Status Banner":{"At the moment we are experiencing issues with the Proton VPN service":["W tej chwili mamy problemy z usługą Proton VPN"],"Learn more":["Dowiedz się więcej"]},"steps":{"Step":["Krok"]},"suggestions":{"Suggestions":["Sugestie"]},"Support":{"Sub category":["Podkategoria","Podkategorie","Podkategorii","Podkategorii"]},"Support article":{"${ readingTime } min":["${ readingTime } min","${ readingTime } min","${ readingTime } min","${ readingTime } min"],"Category":["Kategoria","Kategorie","Kategorii","Kategorii"],"Didn’t find what you were looking for?":["Nie znaleziono szukanej frazy?"],"General contact":["Skontaktuj się, wykorzystując ogólne zapytanie"],"Get help":["Uzyskaj pomoc"],"Legal contact":["Skontaktuj się z działem prawnym"],"Media contact":["Skontaktuj się z zespołem medialnym"],"Partnerships contact":["Skontaktuj się z zespołem ds. partnerstwa"],"Reading":["Odczytywanie"]},"Support Form Platform option":{"VPN for Android TV":["VPN dla Android TV"],"VPN for Apple TV":["VPN na Apple TV"],"VPN for Chromebook":["VPN dla Chromebooka"]},"Support troubleshooting":{"App version":["Wersja aplikacji"],"Browser":["Przeglądarka"],"Check if this helps":["Sprawdź, czy to pomoże"],"Choose a product":["Wybierz produkt"],"Did this solve your issue?":["Czy Twój problem został rozwiązany?"],"Faster assistance is just a few clicks away":["Szybsze wsparcie uzyskasz w kilka kliknięć"],"How can we help?":["Jak możemy pomóc?"],"No, contact support":["Nie, skontaktuj się z pomocą techniczną"],"Please fill out one field after another":["Wypełnij pola jedno po drugim"],"Please make your selections":["Dokonaj wyboru"],"Proton account":["Konto Proton"],"Proton for Business":["Proton for Business"],"Thank you for your feedback":["Dziękujemy za podzielenie się opinią"],"What can we help with?":["W czym możemy pomóc?"],"Yes":["Tak"]},"support_modal_search_query":{"Search query":["Zapytanie wyszukiwania"]},"support_search_button":{"Search":["Szukaj"]},"support_search_i_am_looking_for":{"I'm looking for":["Szukam"]},"SupportForm":{"For a faster resolution, please report the issue from the Bridge app: Help > Report a problem.":["W celu szybszego rozwiązania problemu dokonaj zgłoszenia przez aplikację Bridge: Pomoc > Zgłoś problem."],"Information":["Informacje"]},"SupportForm:option":{"Account Security":["Bezpieczeństwo konta"],"Contacts":["Kontakty"],"Custom email domain":["Niestandardowa domena e-mail"],"Email delivery and Spam":["Dostarczanie wiadomości e-mail i spam"],"Encryption":["Szyfrowanie"],"Login and password":["Login i hasło"],"Merge aliases and accounts":["Scalanie aliasów i kont"],"Migrate to Proton":["Migracja do Proton"],"Notifications":["Powiadomienia"],"Other":["Inne"],"Plans and billing":["Plany i rozliczenia"],"Proton for Business":["Proton for Business"],"Sign up":["Rejestracja"],"Storage":["Przestrzeń dyskowa"],"Users, addresses, and identities":["Użytkownicy, adresy i tożsamości"]},"SupportForm:optionIntro":{"Select a topic":["Wybierz temat"]},"Testimonial":{"Awards":["Nagrody"],"Customers":["Klienci"],"Featured":["Wyróżnione"],"Go to testimonial source":["Przejdź do źródła referencji"],"Open source of award":["Otwórz źródło nagrody"],"Open source of quote":["Otwórz źródło cytatu"],"Reviews":["Opinie"],"Videos":["Pliki wideo"],"Watch on TikTok":["Oglądaj na TikTok"],"Watch on YouTube":["Oglądaj na YouTube"]},"TestimonialCategory":{"Awards":["Nagrody"],"Customers":["Klienci"],"Featured":["Wyróżnione"],"Media":["Multimedia"],"Reviews":["Opinie"],"Videos":["Pliki wideo"]},"Text":{"If you need help, check out our ${ supportLink }.":["Jeśli potrzebujesz pomocy, sprawdź naszą ${ supportLink }."],"The page you’re looking for might have been removed, or it could be an\nold link.":["Strona, której szukasz, mogła zostać usunięta lub link jest przestarzały."],"Your question may already have an answer in our knowledge base:":["Na Twoje pytanie może już istnieć odpowiedź w bazie wiedzy:"]},"Title":{"On this page":["Na tej stronie"],"Related articles":["Powiązane artykuły"],"Share ${ thisPage }":["Udostępnij ${ thisPage }"],"Switch to Proton Pass - Contact us":["Przejdź na Proton Pass – skontaktuj się z nami"],"Thank you!":["Dziękujemy!"],"this page":["tę stronę"]},"tooltip_vpn":{"Access blocked content and browse privately. Includes ${ TOTAL_VPN_SERVERS }+ servers in ${ TOTAL_VPN_COUNTRIES }+ countries, highest VPN speed, ${ TOTAL_VPN_CONNECTIONS } VPN connections, worldwide streaming services, malware and ad-blocker, and more.":["Uzyskaj dostęp do zablokowanych treści i przeglądaj prywatnie Internet. Obejmuje ponad ${ TOTAL_VPN_SERVERS } serwerów w ponad ${ TOTAL_VPN_COUNTRIES } krajach, najwyższe prędkości VPN, ${ TOTAL_VPN_CONNECTIONS } połączeń VPN, usługi streamingu na całym świecie, blokowanie reklam oraz złośliwego oprogramowania i wiele więcej."]},"vpn_servers":{"Get Proton VPN Plus":["Wybierz plan Proton VPN Plus"]},"wallet_signup_2024:Action":{"Get Proton Wallet":["Wybierz Proton Wallet"]},"wallet_signup_2024:Homepage hero product link title":{"Wallet":["Portfel"]},"wallet_signup_2024:Homepage product navigation bar":{"Wallet":["Portfel"]},"wallet_signup_2024:menu item":{"Bitcoin guide":["Przewodnik po systemie Bitcoin"],"Proton Wallet news":["Aktualności Proton Wallet"],"Proton Wallet support":["Wsparcie dla Proton Wallet"]},"wallet_signup_2024:Pricing":{"Includes everything in Proton Unlimited and":["Zawiera wszystko, co Proton Unlimited oraz"],"Limited availability":["Ograniczona dostępność"],"The easiest way to securely own, send, and receive Bitcoin":["Najprostszy sposób na bezpieczne przechowywanie, wysyłanie i odbieranie waluty Bitcoin"]},"wallet_signup_2024:ProductRange":{"Discover Proton Wallet":["Odkryj rozwiązanie Proton Wallet"],"Store and transact Bitcoin privately with an encrypted self-custody wallet.":["Bezpiecznie przechowuj i przesyłaj Bitcoiny dzięki szyfrowanemu portfelowi typu self-custody."]},"wallet_signup_2024:wallet bitcoin":{"Learn about Bitcoin, the Internet's value network.":["Dowiedz się więcej o internetowym systemie gotówkowym Bitcoin."]},"wallet_signup_2024:wallet overview":{"Ensure you're always in control of your Bitcoin.":["Miej pełny nadzór nad swoimi Bitcoinami."]},"wallet_signup_2024:wallet security":{"The encrypted, open-source wallet that puts you in control.":["Szyfrowany portfel o otwartym kodzie źródłowym, który daje Ci pełną kontrolę."]}}},"base":"blog","cdn":{"enabledForAssets":true,"enabledForImages":true,"url":"https://pmecdn.protonweb.com/"},"unleashApi":"https://account.proton.me/api"};
window.frameworkContext = frameworkContext;
const context = frameworkContext.base === '' ? '' : `${frameworkContext.base}/`;
window.__toAssetUrl = (filename) => {
if (frameworkContext.cdn !== undefined && frameworkContext.cdn.enabledForAssets === true) {
return `${frameworkContext.cdn.url}${context}${filename}`;
} else {
return `/${context}${filename}`;
}
};
})();
Analysis of Proton’s July service disruption | Proton
Starting Monday, July 11, and ending Wednesday, July 13, Proton Mail, Proton VPN(new window), and Proton Drive experienced intermittent service disruptions, some of which affected some users for an hour or more. These resulted from an unexpected error, not an attack or other malicious activity.
This doesn’t meet the standards we hold ourselves to, nor is it what the Proton community expects from us. We apologize to you, and we’ve taken steps to make these types of interruptions much less likely in the future. Below we explain what happened, how we stabilized the situation, and what we’ve done to prevent future disruptions.
Background
Over the last several months, our database team has been upgrading our relational databases to be more reliable, faster, and more scalable. We’ve extensively tested these upgrades and, up to this point, performed dozens of them without incident.
We finished the last upgrade on the morning of Sunday, July 10. We saved this particular database for last because it’s the ultimate source of truth for community member account and email address information. It’s also very, very busy. We had identified this database’s high usage rate as a risk. We already had several initiatives in progress to reduce its workload and improve performance to make the overall system more resilient and scalable.
We decided to upgrade the database before these initiatives were complete because the extensive testing and our experience from the previous database upgrades indicated the new database would be faster. As part of this upgrade, we also moved the database to a newer, faster server. We believed this combination of newer software and hardware would improve performance and buy us additional margin to safely implement our more invasive database optimizations.
The incident
All services and metrics were normal until Monday, July 11, at 2:35 PM UTC. As traffic increased, new connections to the new database began to fail, activating automatic protective measures that prevented new connections. We raced to figure out what was wrong and reduce the database’s load by turning off optional or low-priority services, like message notifications.
Usually, if an issue like this arises, we would simply undo the update and revert to the previous software version. Unfortunately, this particular upgrade was irreversible as it involved changing the database’s data formats, and we’d already recorded more than 24 hours of changes using the new version. That meant we were on the clock to mitigate the symptoms we observed, find the root cause, and find a permanent way forward.
We now know that the database software was faster after the upgrade, but the new connections to that database were not. Part of this additional connection latency was inherent to the new database codebase, but each new connection also had an extra round-trip network communication, increasing the strain on an already busy networking stack.
This extra round-trip communication was caused by a new authentication default introduced in a recent patch of the database software. This may not sound like much, but this database processes so many connections that the two extra packets the new authentication process added and the additional inherent connection latency were enough to overwhelm the server on both the MySQL and kernel network levels.
Our response
By the end of Monday, we hadn’t discovered these extra packets, so while we continued to investigate, we also worked to reduce the database’s connection rate. The steps we took included:
Shifting more read-only workloads away from the writable database server
Additional caching of objects and common queries where possible
Deferring low-priority mail to smooth out delivery spikes
We confirmed the authentication issue on Wednesday, July 13, at 1 AM UTC. To mitigate it, our team worked to bring new servers online, which we used to spread out the load over multiple servers to prevent any single one from being overwhelmed.
At 8:42 AM UTC, we changed the authentication parameter back to the default used in the previous version. This helped reduce the activity load on the database server and, along with the optimizations already made, essentially eliminated the errors and alerts we had received the last two days.
However, we discovered a secondary issue at 2:14 PM UTC that began when we spread the workload over multiple servers. These new replica servers were dedicating more than 50% of their processing power to verify their synchronization with the writeable primary database. This meant that at peak activity times, the number of connections would overwhelm the replica servers, causing the traffic to be rerouted back to the main database, which in turn created instability and occasionally interrupted service until activity levels dropped.
We eliminated this synchronization load (by caching) shortly before 4:00 PM UTC and stabilized the replica databases, which permanently resolved the intermittent instability.
Going forward
In the days following the incident, we developed, validated, and executed the first of several planned splits of this database to permanently reduce its workload. Our team implemented these splits successfully without disrupting our service. We also have initiatives in progress to improve our connection pooling so that this specific problem cannot reoccur in the future.
These measures, while necessary, are insufficient. They make us better prepared to fight the last war, but they do not anticipate future problems or address the decision-making process that led to this incident.
To achieve that goal, our infrastructure and application teams are performing a thorough multi-stage review of all services and systems to better understand possible failure modes and how we can mitigate them. The reviewers consist of service owners and other team members to ensure we have subject-matter expertise and fresh sets of eyes. The emphasis of this review is to prevent failures but also to localize potential failures and prevent cascades and large-scale service interruptions to the extent possible. Some fixes will be quick, and others are architectural and will take time, but we’re committed to making Proton services as reliable as the Proton community expects and deserves.
On the decision-making side, we’ve dissected the process and inputs that led to the decision to do the upgrade before the split to ensure that we make the correct decision next time. Very, very few changes we make, whether to infrastructure or the application code, are irreversible, and for good reason. In fact, this is the only such change in the last several years. In this case, attempting to make the change reversible would not have been feasible. But the fact that it was irreversible should have triggered a more cautious change approval process, and the upgrade’s previously successful track record made us overconfident that this database would behave the same, despite its vastly heavier workload.
This is an opportunity for us to re-evaluate our infrastructure approach, and ultimately it will lead to us being more resilient and better prepared in the future. Thank you to everyone in the Proton community for your patience during the service disruption. We have learned many lessons that will serve us well as we work to build an internet where privacy is the default, and we thank you again for your support.