It was 12.40am on Wednesday, March 10, 2021. Until that moment, SBG2 was an unremarkable data center on a not-very-interesting industrial site in Strasbourg, a small city best known for its role as a hub of European bureaucracy.
SBG2 sported a large OVH logo (its owner’s full name is OVHcloud) alongside the kind of colorful cladding that often cheers up a facility whose facade has no other distinguishing features. Its five stories provided 500 sqm (5,380 sq ft) of data center space on the West bank of the river Rhine, where France borders Germany.
It’s not clear precisely what happened next, but within six hours SBG2 would be the most talked-about data center in the world. It would also no longer exist.
Before dawn broke, fire had ripped through the building. Its bright cladding was a misshapen, smoldering shell. The 30,000 servers inside were destroyed, and the adjacent SBG1 was terminally damaged. Data was lost, and companies across the Internet were reeling. At the height of the fire, 3.6 million websites corresponding to 464,000 domain names were unavailable.
No one was hurt. But, in terms of property, this was the biggest single disaster ever to hit a data center. Never before or since has a complete data center burnt down.
The sector exists to provide reliable storage for data, and prides itself on detailed preparations for every disaster which might happen. Data centers do not fail. But this one did more than that, it completely self-destructed.
It is clear that something at SBG2 went badly wrong.
Eerie silence
In other industries, we would know what happened. When planes crash, “black box” flight recorders tell the story. When companies go under, forensic accountants reconstruct the process in detail.
In striking contrast to that, the data center sector, which prides itself on the precision of its operations, has been silent over SBG2.
Three years on, there is no official story of its final hours. OVHcloud has cleared away the rubble, built a new data center on the site, and ignored any requests for information about it. In the aftermath of the fire, hundreds of OVHcloud customers threatened to sue. Yet in three years, DCD is aware of only two customers who have reached the courts. Both were awarded compensation, but neither has seen any money yet, as the cases are being appealed.
The secrecy will come as no surprise to data center people. Covering up failures is normal practice and, as industry observers have commented time and again, it is supremely bad practice - because it means the same thing will very likely happen again.
Data center outages are often “recurring failures,” consultant Ed Ansett of i3 Solutions, told DCD in 2017. Ansett has been consulted on data center failure incidents for years, and told us that many have the same root causes. The failure data is hidden by non-disclosure agreements, and the mistakes are repeated.
”I came to the conclusion some time ago that people were not learning from experience,” said Ansett.
Days after an outage at a British Airways data center grounded thousands of flights, Ansett and colleagues announced DCIRN, a “crash report” service to share data center incident data, learn from mistakes, and avoid repetitions.
In the days after SBG2 burnt, Ansett and his DCIRN colleague Dennis Cronin shared lists of data center fires, and expressed the hope that maybe this fire would be the one that persuaded the data center industry to finally open up.
Three years on, that hope has been dashed. The SBG2 fire has never been fully explained. And DCIRN itself was wound up in August 2021. “We were unable to attract sponsorship, so we’ve put the initiative on hold,” Ansett told us in 2021, on the first anniversary of the SBG2 fire.
Despite this, we can piece together a lot of facts about the fire. And they do not make for comfortable reading.
What is OVHcloud?
But first, why should we care about OVHcloud? As the name suggests, it operates in the cloud, and has some similarities to market leader AWS (Amazon Web Services). It now has facilities in the US, Singapore, Australia, and India, but it has a very different character from the other cloud players.
It is on a much smaller scale, it is based in Europe, with its headquarters in Paris, and it has made its own engineering choices at every stage, talking openly about them.
OVH was founded in 1999 by Octave Klaba, a 24-year-old computer science student at the Icam University in Lille, who spotted the demand for Internet hosting, and borrowed around $3,000 (25,000 francs in France’s pre-Euro currency).
Klaba’s family had migrated from Poland to France after the fall of the Berlin Wall; his parents Henryk and Halina helped with the loans, and his brother Miroslav joined the initiative. The company started with rented servers in a Paris data center, but quickly took on its own 3,000 sqm (32,390 sq ft) space in Paris in 2003, before expanding into a derelict industrial site in Roubaix, northwest of Paris in 2005.
OVH focused on building things in-house. As far back as 2001, Klaba was building casings and chassis, and in-house “GreenBox” servers. In 2003, he added his own design of water-cooling system.
When OVH moved to the Roubaix complex, Henryk Klaba ran a metal workshop to make OVH’s casings and racks - using a horizontal rack design unique to the industry. That was still in operation in 2019, when DCD visited.
The company’s liquid-cooled racks make lower demands on the building chillers, so the company was able to expand piecemeal within Roubaix’s warren of disused factories, rather than building new high-tech facilities.
But in 2012, the company expanded with a new data center in Canada, and its first facility on the Strasbourg site. By then, OVH had proudly announced EcoRoom, a data center design that did away with air-conditioning in favor of “a natural ventilation system that uses hot and cold airflow.”
We will come back to that aspect.
During this period, OVH was a geek-hosting company, with geek customers. They loved to see Klaba jamming with musicians onstage at the annual company show. OVHcloud picked up some major customers, but also had a reputation for supporting the more anarchic end of the industry, including gamers, cryptocurrency firms, and both good and bad hackers. Around 2010, Julian Assange’s Wikileaks site found its home there.
Gaining respectability
But by 2021, when SBG1 burnt, things were changing, and the blackened servers held a snapshot of a company in transition.
They held tens of thousands of OVH’s traditional customers: small businesses, online fora, and the dystopian survival game Rust. Security researchers at Kaspersky commented sardonically that a lot of APT security threats went down with the building.
Other sites that went down included the Centre Pompidou Arts Center in Paris, several French government departments, the UK’s Vehicle Licensing Agency, and the European Space Agency. These were signs that OVHcloud had taken on new kinds of customers, taking on a new identity as the web became more accepted.
In 2019, the company changed its name to OVHcloud and appointed a new CEO - Michel Paulin - brought in from the telecoms industry, where he had previously led French telco SFR.
Klaba was still in place, as founder and CTO, and he still owned 80 percent of the company. But it was becoming a corporate European champion, eyeing a slice of the emerging cloud industry.
OVHcloud was by no means a giant. It is likely the biggest public cloud provider outside the US and China, but it sits a long way beyond AWS, Azure, and Google Cloud, who own two-thirds of the market between them. Most sources put OVHcloud behind Oracle, IBM (Kyndryl), and China’s Alibaba, Huawei, and Tencent. It may have as much as one percent of the world’s cloud business, maybe a bit more.
But public sector agencies were selecting it as a way to support Europe’s native tech sector, and also to take a step towards digital sovereignty - storing European citizens’ data on servers within Europe, where it is protected by Europe’s GDPR privacy legislation.
The new OVHcloud was winning partnerships with regional services companies including Deutsche Telekom, Atos, and Capgemini.
OVHcloud was reaching a new level of respectability, and it was about to take the next step. On March 8, OVHcloud launched the process for a stock market flotation.
Two days after that, SBG2 burnt.
What happened?
Despite the secrecy, we know quite a lot about the fire. A hundred firefighters, with 44 firefighting vehicles, took six hours to bring the blaze under control. At the height of the blaze, Strasbourg's French fire department called in a Franco-German pump ship operating on the Rhine.
We know that the fire started in the power supply room. Water may have been involved, along with a large number of batteries intended to provide continuous power in the event of a failure of the grid. That much emerged in the first few days after the fire was started.
One year on from the fire, the Bas-Rhin firefighters’ report was published. It was later taken down from the Internet, but not before commentators had noted the shocking details, many of which were confirmed in an investigation by the Bureau of Investigation and Analysis of Industrial Risks (BEA-RI).
Firefighters arrived at the scene one minute before 1am. They found electrical arcs more than one meter long flashing around the door to the power room. "The flashes were impressive and the noises deafening," their report said.
The later BEA-RI report had access to CCTV footage and sensor data which showed the moment the fire started in the power room. It also noted that there were high moisture readings near one of the inverters in a power room during the hour before the fire started.
The firefighters could see enough to know this was an electrical fire. The first thing they tried to do was switch off power to the building, but that turned out to be more complicated than it sounds.
At 1.13am, firefighters attempted to cut the power supply, but the power still flowed. At 1.30am, they got the local utility to cut power upstream, but there was still electricity going into the building until at least 2.14am, and as late as 03:28am, according to the firefighters’ report.
The BEA-RI report adds some details. Like many data centers, SBG2 had a backup power supply based around a set of diesel generators designed to start up automatically if there was an interruption to the grid. When the utility cut power, the diesels’ control systems fired them up, even though the building was ablaze,
This means that some OVH customers’ servers were still running while the building was ablaze.
The delay in switching off power kept the firefighters out of the power room, and gave the flames time to spread. The power room had a wooden ceiling designed to withstand fire for one hour, and electrical ducts were not insulated. By 1.30am, the fire had reached the rest of the building.
The fire was apparently not impeded by any sprinkler systems or other fire prevention measures, and gained pace in the main building, which had wooden floors and ceilings.
By 2.14am, SBG2 was “totally ablaze”, and the firefighters shifted their efforts to preventing the fire spreading to SBG1, which narrowly escaped complete destruction.
SBG2’s “EcoRoom” design was intended to create a flow of air through the building. When the fire broke out, that airflow became a chimney, and the building became an inferno.
That kind of design might sound bizarre, but EcoRoom was a product of OVHcloud’s tech culture, born from a focus on cost, speed, and efficiency, which evolved on the site.
The first OVH facilities at Strasbourg, SBG1 and SBG4, had been hastily built from shipping containers, and did not feature standard redundancy features. This led to a serious outage in 2017, which took down OVH services, but did not damage the buildings.
SBG2 was an EcoRoom facility, using the idea of “free” cooling, letting waste heat from the servers rise out of the building, and drawing cool air in from outside.
OVHcloud had seemingly not realized that this would create a risk of spreading any fire that got established in the building. The company was also apparently unaware of the multiple dangers noted by the firefighters.
Pivot to silence
In the first days after the fire, Octave Klaba was the earnest, lovable geek he had always been. He was on-site with the clean-up, tweeting, posting videos, and promising support and full transparency.
On March 11, he posted a video with interesting details about how the fire might have started. When firefighters arrived, he said, their thermal imaging cameras showed two power supplies (UPS7 and UPS8) were on fire. That was interesting because UPS7 had received maintenance the day before the fire broke out.
The BEA-RI report would later note some maintenance work: inverters in power room 2 had needed attention because of "recurring problems switching to bypass mode" which were "unexplained."
In his video the day after the fire, Klaba said: "The supplier came and changed a lot of pieces inside UPS7, and they restarted it in the afternoon. It seemed that it was working, but in the morning we had the fire."
Over the next weeks, Klaba tweeted details of the company’s efforts to clean servers, retrieve data, and restart SBG2, and the other facilities on the site, all of which had been shut down. Servers were taken to Roubaix for cleaning, and racks in SBG1, SBG3, and SBG4 were powered up one by one.
On Tuesday March 16, 2021, less than a week after the fire, OVHcloud handed over electrical equipment to police investigators.
Klaba was treating the incident as a technological challenge but the customers didn’t all see things the same way. Many believed they had been paying for a reliable service with backups, and were shocked that their data was lost.
Klaba promised that all customers would have free backups by default in future, and promised a big change: "I believe this incident will change the standards of the industry" he said.
He also said he would reveal the cause later that week, on Friday, March 19.
Neither of those promises came true.
On the Friday, a second small fire took place in the battery room of the damaged SBG1. Over the weekend, OVHcloud abandoned efforts to restart that crippled facility - and Klaba began to clam up.
The OVHcloud founder continued to tweet details of the gradual return of services from Strasbourg. A live display on the company’s site showed servers coming back to life.
But he stopped talking about the fire, and started to make promises about the future, including a plan for a laboratory to lead industry research into data center fire prevention.
The change seems to have been driven by his CEO. Klaba’s tweets and videos increasingly talked about decisions made “with Michel [Paulin].”
The aftermath: legal quagmire
The change in attitude became clearer one month after the fire, when OVHcloud published its offers of compensation for affected customers.
For many, it was a disappointment. OVHcloud was offering refunds for lost service, rather than compensation for business losses. Customers were not even compensated for lost data, unless they had paid for backups, with the company saying cloud customers should be handling their own disaster recovery plans.
Customers with virtual private servers got a refund for the months when service was not available. If the server had been destroyed they got another six months, and if data was lost… well, that depended on whether they had been paying OVHcloud for backups.
Larger customers would have a more customized plan, as they would have to have their services rebuilt or migrated from any servers that could be restarted. Klaba was still hands-on and accessible for these people, tweeting: "We‘ve created the migration process included copy of PBs of data from SBG to RBX. Going fast but we need to talk with each cust[omer,] DM me."
That didn’t satisfy everyone. A number of people sued OVHcloud for negligence, and Paris law firm Ziegler Associates started gathering names for a class action suit - an unusual process that has only recently been introduced in French law.
That class action appears never to have been delivered. Ziegler was in regular touch with DCD during 2022 as the number of plaintiffs climbed to 140, seeking a total of €10 million.
In June of 2022, Ziegler told DCD it had a formal letter ready to send to OVHcloud, as the start of a negotiation process. OVHcloud never acknowledged receipt of the letter, and both Ziegler and OVHcloud stopped answering our questions in July of 2022.
The Ziegler class action, which may or may not have been secretly settled, was based around negligence over fire protection, which appeared to be revealed by the firefighters’ report. Before it ceased to answer our questions, OVHcloud said that report was not valid, because the firefighters did not understand data centers. It promised its own report would give a fuller explanation, but that report never emerged.
Meanwhile, up to four firms were suing OVHcloud individually over a slightly different issue. These were companies who believed OVHcloud had backed up their data safely - but lost it anyway.
Two of these cases reached court in 2022, and OVHcloud lost both. The court’s verdict was utterly damning on its backup procedure.
The plaintiffs, Bati Courtage and Bluepad, had both paid OVHcloud extra to safely backup their data - but learned after the fire that the company had not done so. The backup copies of their data were in the same data center as the live servers, and both were lost in the fire.
The Commercial Court of Lille Métropole ruled in February 2023 that this did not deliver the service OVHcloud had promised, and ordered it to pay €100,000 and €150,000, respectively, to Bati Courtage and Bluepad
When it emerged that the property firm Bati Courtage’s backup was lost, OVHcloud’s representatives attempted to argue that keeping a backup in the same room as the primary data was perfectly okay, in the letter of their backup contract. This argument was given short shrift by the judge, who pointed out that the service provider had plenty of other data centers to keep the backups in.
The case of SaaS software vendor Bluepad was a bit different. The company paid OVHcloud for extra servers and performed its own backups. Bluepad had access to OVHcloud’s management console, which showed its primary data was in SBG1, and the backup was in SBG2.
After the fire, Bluepad was shocked to discover that the console was lying. Both servers were in SBG2, and both were dead. OVHcloud argued that the console information was just an indication, but the judge slapped that down.
Was OVHcloud lying or incompetent about Bluepad’s data? We don’t know, but we know one thing more. The data could have been saved, because OVHcloud retrieved an undamaged Bluepad hard drive from the wreckage. The data was still there - until OVHcloud broke its own procedures and switched the drive on, at which point it formatted itself and the data was lost.
One year on, these findings have not led to others. OVHcloud has appealed, and the matter is sub judice again. The results are sealed until the appeal is heard. DCD has spoken to French lawyers who warn that the country’s legal system could take a very long time to settle these cases finally, let alone any others.
A French contact has told DCD that: “In France, justice deadlines are very long.”
Commentators suspect that the process has been deliberately stalled, in deference to OVHcloud’s status as a national champion, which has had plenty of support from the French government in the past.
OVHcloud and Klaba bounce back
Meanwhile, OVHcloud seems to have forged on completely unaffected by the disaster. It ended 2021 with a successful stock market flotation which raised some €350 million ($380m) and valued the company at well over €4 billion ($4.36bn).
In 2022, however, the company reported a loss because of the fallout from the fire.
The value of the company has since fallen back to around €2 billion ($2.18bn), but its role as a European champion is as bankable as ever. In 2022 it had a loan of €200 million ($218m) from the European Investment Bank.
It has carried on building new data centers, including a new (and hopefully fireproof) facility on the site of SBG2.
SBG5 is the first new OVHcloud data center based around a new design intended to prevent a repeat of the SBG2 disaster. The building has 1,700 sqm (18,300 sq ft) of data center space in 19 isolated rooms separated by masonry walls which “offer fire resistance of up to two hours.”
OVHcloud also says it has a gas fire extinguishing system, and has built the data center’s seven energy rooms and three battery rooms outside the building.
The company has kept up with hot trends in the industry, offering an AI training service, as well as hosting a quantum computer from the French IT consultancy Atos.
Despite the engineering failure at SBG2, it has burnished its credentials in data center technology, showing off a further evolution of its liquid cooling system, which seals individual servers in their own immersion tanks, which are penetrated by pipes connected to the existing water cooling system.
Octave Klaba is still visible at all of OVHcloud’s events. But even after the IPO, the Klaba family owns some 70 percent of the company, and Octave has been spending some of his money as an entrepreneur elsewhere.
Even before the fire, Klaba was in the process of buying the gaming firm Blade, which ran the Shadow cloud gaming service. He added a Dropbox-style storage service to Shadow.
In 2023, he bought a French search engine, Qwant, and started making grandiose promises. Just as OVHcloud wants to be a European AWS, Klaba said in interviews that he could grow Qwant into a European Google.
In 2023, Qwant had spent €50 million in public funds, and was on its last legs. Klaba has bought it, and merged it into a new SaaS company called “Synfonium” which he has promised will match Google service for service, with European technology.
Neither OVHcloud nor Klaba will talk about the SBG2 fire now, but they are both using the experience in surprising ways, having seen that national champions get special treatment.
The information seems to show OVHcloud running a dangerous and unreliable data center, and watching it go up in flames. It is not repentant. If anything, it is proud of itself, presenting SBG5 as the last word in resilience, springing from the knowledge it gained.
OVHcloud knows that France is prepared to forgive and conveniently forget the failings of its most important businesses. It has seen that effect with another anointed French tech leader, services company Atos.
It looks like both OVHcloud and Klaba have learned a lesson: being a European tech champion makes you fireproof.
DCD reached out to OVH for this piece, but received no reply.
