We want to provide some additional information on the power issue in a single Availability Zone in the ME-CENTRAL-1 Region. At around 4:30 AM PST, one of our Availability Zones (mec1-az2) was impacted by objects that struck the data center, creating sparks and fire. The fire department shut off power to the facility and generators as they worked to put out the fire. We are still awaiting permission to turn the power back on, and once we have, we will ensure we restore power and connectivity safely. It will take several hours to restore connectivity to the impacted AZ. The other AZs in the region are functioning normally.
Most likely, those were debris from the interception of missiles flying overhead and being destroyed on their way to a military target.
AFAIK, there have been no confirmed signs of civilian sites being targeted directly, and it would also be unlikely that actual missiles would cause so little damage that you could patch your datacenter up and get it ready to go within hours.
Thanks for the links, which I've reviewed. Allow me to clarify: I meant sources that confirm that the civilian places hit (eg. hotels and residential buildings) were the actual targets.
Local and official news all say that these were hit by debris from intercepted missiles/drones (on their way to somewhere else). There is a major difference between this, vs. if those buildings were directly being targeted.
AFAICT your linked sources indicate that the oil installations and ports were targets, but not the hotels and buildings.
I'm asking in good faith as this makes a significant difference.
I don't see the large difference between a civilian port, a civilian oil facility or a civilian aluminum factory vs a hotel on the topic of whether the Iranians are capable of targeting a civilian data center, however, assuming you are curious, here goes
Finding these take time so I am sorry if this is going to be the last of these sources I'll paste, for example Bahrain luxury apartments building being hit:
This reminds me of a visit to an Equinix data centre where the sales person was droning on and on about how incredibly reliable their power supplies were, how uninterruptible everything was, etc, etc…
Essentially, he was trying to assure us that no-no-no, we don’t need multiple zones like the public clouds, they can instead guarantee 100% uninterrupted power under all circumstances.
A bit bored and annoyed, I pointed to the giant red button conspicuously placed in the middle of a pillar and asked what it is for.
“Oh, that’s in case there’s a fire!”
“What does it do?”
“It cuts… the power… uhh… for the safety of the fire department.”
“So… if there’s a wisp of smoke in a corner somewhere, the fireys turn up, the first thing they do is… cut the power?”
Equinix in Sydney plonked 2 datacenters right on top of each other, and still insists that they are useful as redundant sites.
There was a locally very funny situation for a while when a tech influencer was insisting both equinix sites could be shut down by a single building collapse. He was wrong, but he wasn't so wrong that people shouldn't make better infrastructure decisions.
The building in question wasnt really tall enough. And would have to be precision demolished to collapse in the way he was afraid of.
It would still cause chaos and possible power issues.
Needs to be taken in context with some Sydney buildings having maintenance defects a few years after they open. Largely due to inferior materials imported from china. The building in question developed some cracks in supporting beams and was briefly evacuated. There was never a chance it was going to topple on its own in a way that impacted more than 1/2 datacenters, so he pivoted to possible terrorism, but even thats largely nonsensical.
I just went hunting for the case and couldnt find it. The gentleman in question had published the claim to his business that was it happens trying to build contacts with defense and intelligence agencies for third party threat assessment. As far as I can tell the business no longer exists and he has deleted their footprint.
But he also posted the claim on public mailing lists so I can probably trawl it up if necessary.
My boss at my first job hit the Big Red Button by swinging his arms too big in our datacenter one day, shutting down hundreds of servers and the mainframe, wreaking havoc for days!
That was when we installed the Big Clear Button Cover.
> we will ensure we restore power and connectivity safely
this would require human intervention and I am a bit worried what if the strike can happen again and human lives might be lost.
IIRC there have been cases in history where sometimes a same location is targeted across multiple days. Obviously, AWS might have local employees working in the region but would there be an evaluation of this threat itself within the relevant team in AWS. What if they try to bring the service back but then missiles are struck again and what if human lives might be lost on it. Let's just hope that it could be part of a evaluation as well.
But I mean,are the employees safe at home?
I guess if the really targeted the data center then home is safer, but in the fog of war maybe the data center wasn't the target?
> But I mean,are the employees safe at home? I guess if the really targeted the data center then home is safer, but in the fog of war maybe the data center wasn't the target?
My gut feeling says that they would be safer at homes than at datacenters. The only large info I have heard is attack on hotels, this datacenter etc. (atleast till right now).
> but in the fog of war maybe the data center wasn't the target?
We can't say this for sure but even if that was the case, I do think that they would see some damage was caused and then, try to double tap it for even more damage. So chances are, even if it wasn't the target previously, it might be the target now?
100% absolutely but its a bit worrying if in the future multiple AZ/datacenters could be start to get targeted?
attacking datacenters within a particular region so that service would have a hard time.
I guess someone can use some other regions DC to have more than (regional?) AZ but for mission critical infra, I can see that having sometimes issue too and you genuinely can't predict any of all of this.
That being said, There should be more than one AZ reliance but IMO also off-site or multi-cloud backups should also be preferred/used as well.
> What about local hospitals which may have service from that data center? There are heroes needed everywhere, all the time.
Off-site backups/Multi Cloud Strategy while encrypting data (and keeping the key safe, key point) might be a better strategy for such mission critical infrastructure.
I'm sure bezos will be really happy someone is being a hero for him in a war zone while he sails his newest yacht to wherever the new version of the island is.
on second thought there is a difference between restoring critical infrastructure in times of crisis vs restoring bot infrastructure for indian spamming operations. choose wisely
Mar 01 9:41 AM PST
We want to provide some additional information on the power issue in a single Availability Zone in the ME-CENTRAL-1 Region. At around 4:30 AM PST, one of our Availability Zones (mec1-az2) was impacted by objects that struck the data center, creating sparks and fire. The fire department shut off power to the facility and generators as they worked to put out the fire. We are still awaiting permission to turn the power back on, and once we have, we will ensure we restore power and connectivity safely. It will take several hours to restore connectivity to the impacted AZ. The other AZs in the region are functioning normally.