In the early hours of August 27, 2026, Proton experienced a widespread outage that impacted services for a number of users. The root cause was a total failure of the cooling system in our Frankfurt datacenter. While all systems at Proton are redundant and we have enough capacity to endure a complete data center failure, there are a small number of scenarios where the failover can take longer and lead to user-facing disruptions.

Here’s a timeline of what happened, what choices we made during the incident and why, and how it was resolved.

Timeline

Just after 11 p.m. (Central European time) on Wednesday, August 26, a cooling system failure occurred in the main room of our Frankfurt datacenter. At around 11:15 p.m., the temperature started rising from approximately 21.8°C (nominal temperature) to 51.9°C in less than half an hour, with some measurement probes reporting 60°C air temperature in the room. As the temperatures rose, server and networking equipment within the facility started to die one by one.

The user-facing incident began at around midnight on August 27, when the failures escalated to the point that critical redundancy was lost. This occurred when both the primary and backup network switch on a critical rack failed, and this rack unfortunately contained several primary database copies. While almost all Proton systems are redundant and will failover automatically/immediately, primary database failovers are not done automatically without human supervision.

We retain this control out of a desire to avoid so-called “split brain” situations, where a temporary unavailability of a primary database means that the replica copies miss some updates and become de-synced in ways that can be difficult to reconcile later. Furthermore, when a primary database failover occurs, the standard operating procedure is to failover to a replica in the same datacenter for latency and performance reasons. However, the specific nature of the problem meant that this might be ill-advised, since we potentially could be failing over to something that would also go down.

The decisions

At this point, Proton’s on-call engineers needed to make a couple consequential decisions while operating under extreme pressure.

  • Do they prioritize bringing the service back online, or prioritize addressing the cooling problem and saving the hardware inside the datacenter?
  • Should we failover to replicas within the same building Frankfurt (faster and less disruptive, but possibly a temporary fix if the heat could not be brought under control), or failover to Zurich?
  • Do we failover everything or just what is down at the moment? We have contingencies for complete data center failure where things failover fully and mostly automatically rather quickly, but a situation where random servers are dying one by one is not handled well by our failover logic.

Ultimately the rate at which temperatures were rising forced us to prioritize saving the hardware versus bringing services back online. This is not a choice that typically needs to be made, because cooling systems are typically redundant, and the complete loss of cooling is quite rare, meaning that there is quite a bit of time before temperatures become critical. The problem is exacerbated by large increase in server power density in recent years with higher power CPUs and GPUs for AI. As a result, what used to take 3-4 hours to go critical went critical in 20 minutes.

The on-call team therefore focused their attention on communicating with the on-site datacenter operations team to restore cooling while powering off as many servers as possible to protect them. Due to a server equipment shortage tied to the ongoing AI boom, lots of this equipment — if lost — would not be possible to replace on short timelines. Saving it had to be a priority, even at the cost of potentially extending the downtime.

By 00:45 CEST, we were able to restore cooling and temperatures at the facility began to drop, and the on-call team switched focus to service recovery. At this point, we made the decision to failover the primary databases to Frankfurt if a replica was still alive, and to Zurich in cases where there was no replica alive in Frankfurt, to avoid changing our traffic flows too much and possibly creating new instability. This option was selected because we assumed that, now that we had the cooling under control, it would be relatively easy to bring Frankfurt back online and faster than switching over to Zurich.

Unfortunately, this turned out not to be the case. During the incident, many network cards in the Frankfurt infrastructure reached a temperature of 105C (normal operating temperature is 45C), which triggers a special temperature protection mode and causes the network cards to be disabled until there is a cold system reset. Our security posture limits the ability to access the out-of-band controller for our systems, which required us to wake up additional staff to assist with the recovery.

By 01:30 CEST, we were able to get most services back online for most users. However, some less critical systems, such as push notifications or payment processing, were not recovered until around 02:00 CEST.

As we reported during the initial incident report, no emails were lost, but email delivery in both directions was delayed during the incident.

While user-facing services were fully restored, that was not the end of the night for our engineers, in particular the database team. Our infrastructure was left in a highly abnormal state, with some primary databases in Zurich and others in Frankfurt, and several of them operating with reduced redundancy and/or reduced performance. Our team worked through the night to resolve the most pressing of these issues, and the work continued through the day on August 27 to restore full redundancy.

While we were able to save almost all of the infrastructure, some servers unfortunately suffered heat death, and we don’t know yet if the heating incident will impact the lifespan of the surviving equipment.

Root cause and next steps

A subsequent investigation on August 27 traced the root cause of the cooling failure to an air filter replacement on both of the redundant air compressors powering the cooling system. Unfortunately, the datacenter operator performed this operation in the middle of the night, without prior notice, and also failed to communicate the cooling failure when it happened, which dramatically cut down the time we had to respond. We are working closely with the operator to prevent a repeat of this incident.

However, it is also a known limitation of our current database infrastructure that an outage of this type could lead to a longer than normal recovery process. The series of events that led to this incident are highly improbable — yet they happened.

The database resilience work required to address this failure mode is already underway and remains planned for completion by the end of the year. Additional infrastructure capacity, including new datacenter space, is also currently being commissioned and is expected to become available within the next few weeks, which will further reduce our single site dependency.

Unfortunately, this incident occurred before those improvements were fully in place. We are now reviewing where we can safely accelerate the remaining work while maintaining the level of care required for changes to critical database infrastructure.

We recognize that our users expect a very high level of reliability from Proton, and this incident reinforces the importance of completing this work and continuing to raise our resilience standards. We apologize again, unreservedly, to every user who was impacted.