A few days ago, a client’s data center "vanished" overnight.
-
@stefano But what monitors the monitor monitors? We need an audio technician in here, stat!
-
A few days ago, a client’s data center "vanished" overnight. My monitoring showed that all devices were unreachable. Not even the ISP routers responded, so I assumed a sudden connectivity drop. The strange part? Not even via 4G.
I then suspected a power failure, but the UPS should have sent an alert.
The office was closed for the holidays, but I contacted the IT manager anyway. He was home sick with a serious family issue, but he got moving.
To make a long story short: the company deals in gold and precious metals. They have an underground bunker with two-meter thick walls. They were targeted by a professional gang. They used a tactic seen in similar hits: they identify the main power line, tamper with it at night, and send a massive voltage spike through it.
The goal is to fry all alarm and surveillance systems. Even if battery-backed, they rarely survive a surge like that. Thieves count on the fact that during holidays, owners are away and fried systems can't send alerts. Monitoring companies often have reduced staff and might not notice the "silence" immediately.
That is exactly what happened here. But there is a "but": they didn't account for my Uptime Kuma instance monitoring their MikroTik router, installed just weeks ago. Since it is an external check, it flagged the lack of response from all IPs without needing an internal alert to be triggered from the inside.
The team rushed to the site and found the mess. Luckily, they found an emergency electrical crew to bypass the damage and restore the cameras and alarms. They swapped the fried server UPS with a spare and everything came back up.
The police warned that the chances of the crew returning the next night to "finish" the job were high, though seeing the systems back online would likely make them move on. They also warned that thieves sometimes break in just to destroy servers to wipe any video evidence.
Nothing happened in the end. But in the meantime, I had to sync all their data off-site (thankfully they have dual 1Gbps FTTH), set up an emergency cluster, and ensure everything was redundant.
Never rely only on internal monitoring. Never.
@stefano About 15 years ago, the place I worked had a supercomputer. One night, the aircon in the machine room failed. The machine kept computing, and the temperature rose. It rose *quite a lot*.
Sadly, the first thing to fail from the heat was the core switch for the room. You know, the one that handles all of the network for everything in the room. Including the temperature alerts.
It was finally spotted about 8am when the security patrol wondered why the door shutters were so hot.
-
@stefano About 15 years ago, the place I worked had a supercomputer. One night, the aircon in the machine room failed. The machine kept computing, and the temperature rose. It rose *quite a lot*.
Sadly, the first thing to fail from the heat was the core switch for the room. You know, the one that handles all of the network for everything in the room. Including the temperature alerts.
It was finally spotted about 8am when the security patrol wondered why the door shutters were so hot.
@darkling nice story! Unfortunately I had to manage something like that, too (A/C broken - switch dead, etc)
-
@darkling nice story! Unfortunately I had to manage something like that, too (A/C broken - switch dead, etc)
@stefano Fortunately, the only thing that did fail after the aircon was the switch. (And a pair of ear muffs which had been hanging on a metal rail -- they'd melted).
The fire brigade turned up, checked everything, and ran some big positive pressure fans to get airflow through the room from one door to the other to cool everything down.
-
@_elena@mastodon.social @stefano@mastodon.bsd.cafe How do I say "I knew it! I'm surrounded by assholes!" in Italian?
-
@randomized @rhoot @stefano how do you monitor your sleep
-
-
A few days ago, a client’s data center "vanished" overnight. My monitoring showed that all devices were unreachable. Not even the ISP routers responded, so I assumed a sudden connectivity drop. The strange part? Not even via 4G.
I then suspected a power failure, but the UPS should have sent an alert.
The office was closed for the holidays, but I contacted the IT manager anyway. He was home sick with a serious family issue, but he got moving.
To make a long story short: the company deals in gold and precious metals. They have an underground bunker with two-meter thick walls. They were targeted by a professional gang. They used a tactic seen in similar hits: they identify the main power line, tamper with it at night, and send a massive voltage spike through it.
The goal is to fry all alarm and surveillance systems. Even if battery-backed, they rarely survive a surge like that. Thieves count on the fact that during holidays, owners are away and fried systems can't send alerts. Monitoring companies often have reduced staff and might not notice the "silence" immediately.
That is exactly what happened here. But there is a "but": they didn't account for my Uptime Kuma instance monitoring their MikroTik router, installed just weeks ago. Since it is an external check, it flagged the lack of response from all IPs without needing an internal alert to be triggered from the inside.
The team rushed to the site and found the mess. Luckily, they found an emergency electrical crew to bypass the damage and restore the cameras and alarms. They swapped the fried server UPS with a spare and everything came back up.
The police warned that the chances of the crew returning the next night to "finish" the job were high, though seeing the systems back online would likely make them move on. They also warned that thieves sometimes break in just to destroy servers to wipe any video evidence.
Nothing happened in the end. But in the meantime, I had to sync all their data off-site (thankfully they have dual 1Gbps FTTH), set up an emergency cluster, and ensure everything was redundant.
Never rely only on internal monitoring. Never.
@stefano 10+ years ago i started volunteering at a festival. Everything was new that year including the small outdoor racks for the area field routers (Juniper MX80). They barely fit but we managed. The racks were left in the sun in the summer. It was only when we enabled Observium (LibreNMS predecessor) that graphs almost everything it gets from SNMP that we discovered the inlet temperature was getting close to 80 degrees C. #monitorallthethings
-
@stefano 10+ years ago i started volunteering at a festival. Everything was new that year including the small outdoor racks for the area field routers (Juniper MX80). They barely fit but we managed. The racks were left in the sun in the summer. It was only when we enabled Observium (LibreNMS predecessor) that graphs almost everything it gets from SNMP that we discovered the inlet temperature was getting close to 80 degrees C. #monitorallthethings
@lasseleegaard true. I'm using my switch's fan speed to understand if my home office room is too warm
-
@stefano 10+ years ago i started volunteering at a festival. Everything was new that year including the small outdoor racks for the area field routers (Juniper MX80). They barely fit but we managed. The racks were left in the sun in the summer. It was only when we enabled Observium (LibreNMS predecessor) that graphs almost everything it gets from SNMP that we discovered the inlet temperature was getting close to 80 degrees C. #monitorallthethings
@stefano since the racks were designed for outdoor use they were water tight, only had small holes in the bottom for cables and very limited infrastructure for air venting like downward facing holes in the “roof”. They could supposedly float.
-
@stefano since the racks were designed for outdoor use they were water tight, only had small holes in the bottom for cables and very limited infrastructure for air venting like downward facing holes in the “roof”. They could supposedly float.
@stefano We ended up cutting some wide cable pipes at an angle and duct taping it to the router so we covered the air inlet with one pipe and the air exhaust with another pipe. The other end of the ducts were led to the outside of the rack, lifted off the ground and pointed downwards to avoid water. That provided new fresh air and a way to get rid of the hot air. We also fashioned some shadow with a sheet of plywood. The year after we put some smaller equipment in
-
@stefano Fortunately, the only thing that did fail after the aircon was the switch. (And a pair of ear muffs which had been hanging on a metal rail -- they'd melted).
The fire brigade turned up, checked everything, and ran some big positive pressure fans to get airflow through the room from one door to the other to cool everything down.
@darkling @stefano
Ferranti Computer Systems, Cheadle (UK) circa 1982. I was a lowly apprentice, at the time working in the department that oversaw the various VAXen that most of the site used. Three full size machines and a handful of microVAX. Kept cool by *three* massive air conditioner units on the external wall. The server room was always chilly. /cont -
A few days ago, a client’s data center "vanished" overnight. My monitoring showed that all devices were unreachable. Not even the ISP routers responded, so I assumed a sudden connectivity drop. The strange part? Not even via 4G.
I then suspected a power failure, but the UPS should have sent an alert.
The office was closed for the holidays, but I contacted the IT manager anyway. He was home sick with a serious family issue, but he got moving.
To make a long story short: the company deals in gold and precious metals. They have an underground bunker with two-meter thick walls. They were targeted by a professional gang. They used a tactic seen in similar hits: they identify the main power line, tamper with it at night, and send a massive voltage spike through it.
The goal is to fry all alarm and surveillance systems. Even if battery-backed, they rarely survive a surge like that. Thieves count on the fact that during holidays, owners are away and fried systems can't send alerts. Monitoring companies often have reduced staff and might not notice the "silence" immediately.
That is exactly what happened here. But there is a "but": they didn't account for my Uptime Kuma instance monitoring their MikroTik router, installed just weeks ago. Since it is an external check, it flagged the lack of response from all IPs without needing an internal alert to be triggered from the inside.
The team rushed to the site and found the mess. Luckily, they found an emergency electrical crew to bypass the damage and restore the cameras and alarms. They swapped the fried server UPS with a spare and everything came back up.
The police warned that the chances of the crew returning the next night to "finish" the job were high, though seeing the systems back online would likely make them move on. They also warned that thieves sometimes break in just to destroy servers to wipe any video evidence.
Nothing happened in the end. But in the meantime, I had to sync all their data off-site (thankfully they have dual 1Gbps FTTH), set up an emergency cluster, and ensure everything was redundant.
Never rely only on internal monitoring. Never.
@stefano I wonder how they generate a big enough power surge.
-
@EnigmaRotor reading this at lunch in a cafe near my house and I keep chuckling and smiling from ear to ear. @stefano is such a treasure
-
@darkling @stefano
Ferranti Computer Systems, Cheadle (UK) circa 1982. I was a lowly apprentice, at the time working in the department that oversaw the various VAXen that most of the site used. Three full size machines and a handful of microVAX. Kept cool by *three* massive air conditioner units on the external wall. The server room was always chilly. /cont@darkling @stefano
Until one morning I arrived to chaos. One of the aircons had failed, & the others, overstressed, had completely iced up, and the reduced airflow had caused the temperature in the room to rise. It was pretty much the hottest I'd ever encountered anywhere!
Fire doors and internal doors were propped open, to get a bit of airflow, and the blocked air cons turned off. The heat then had a chance to melt the ice, and they could be brought back online later. I think it took all day. -
-
-
@stefano I wonder how they generate a big enough power surge.
-
A few days ago, a client’s data center "vanished" overnight. My monitoring showed that all devices were unreachable. Not even the ISP routers responded, so I assumed a sudden connectivity drop. The strange part? Not even via 4G.
I then suspected a power failure, but the UPS should have sent an alert.
The office was closed for the holidays, but I contacted the IT manager anyway. He was home sick with a serious family issue, but he got moving.
To make a long story short: the company deals in gold and precious metals. They have an underground bunker with two-meter thick walls. They were targeted by a professional gang. They used a tactic seen in similar hits: they identify the main power line, tamper with it at night, and send a massive voltage spike through it.
The goal is to fry all alarm and surveillance systems. Even if battery-backed, they rarely survive a surge like that. Thieves count on the fact that during holidays, owners are away and fried systems can't send alerts. Monitoring companies often have reduced staff and might not notice the "silence" immediately.
That is exactly what happened here. But there is a "but": they didn't account for my Uptime Kuma instance monitoring their MikroTik router, installed just weeks ago. Since it is an external check, it flagged the lack of response from all IPs without needing an internal alert to be triggered from the inside.
The team rushed to the site and found the mess. Luckily, they found an emergency electrical crew to bypass the damage and restore the cameras and alarms. They swapped the fried server UPS with a spare and everything came back up.
The police warned that the chances of the crew returning the next night to "finish" the job were high, though seeing the systems back online would likely make them move on. They also warned that thieves sometimes break in just to destroy servers to wipe any video evidence.
Nothing happened in the end. But in the meantime, I had to sync all their data off-site (thankfully they have dual 1Gbps FTTH), set up an emergency cluster, and ensure everything was redundant.
Never rely only on internal monitoring. Never.
Oh my
-
A few days ago, a client’s data center "vanished" overnight. My monitoring showed that all devices were unreachable. Not even the ISP routers responded, so I assumed a sudden connectivity drop. The strange part? Not even via 4G.
I then suspected a power failure, but the UPS should have sent an alert.
The office was closed for the holidays, but I contacted the IT manager anyway. He was home sick with a serious family issue, but he got moving.
To make a long story short: the company deals in gold and precious metals. They have an underground bunker with two-meter thick walls. They were targeted by a professional gang. They used a tactic seen in similar hits: they identify the main power line, tamper with it at night, and send a massive voltage spike through it.
The goal is to fry all alarm and surveillance systems. Even if battery-backed, they rarely survive a surge like that. Thieves count on the fact that during holidays, owners are away and fried systems can't send alerts. Monitoring companies often have reduced staff and might not notice the "silence" immediately.
That is exactly what happened here. But there is a "but": they didn't account for my Uptime Kuma instance monitoring their MikroTik router, installed just weeks ago. Since it is an external check, it flagged the lack of response from all IPs without needing an internal alert to be triggered from the inside.
The team rushed to the site and found the mess. Luckily, they found an emergency electrical crew to bypass the damage and restore the cameras and alarms. They swapped the fried server UPS with a spare and everything came back up.
The police warned that the chances of the crew returning the next night to "finish" the job were high, though seeing the systems back online would likely make them move on. They also warned that thieves sometimes break in just to destroy servers to wipe any video evidence.
Nothing happened in the end. But in the meantime, I had to sync all their data off-site (thankfully they have dual 1Gbps FTTH), set up an emergency cluster, and ensure everything was redundant.
Never rely only on internal monitoring. Never.
@stefano There was an attack a few years back near here where they dropped burning rubbish into manholes around a a data centre; the theory at the time was it was to try and cut off some CCTV or alarm monitoring for something. Well caught!
and definitely at the very least a cameo with a line from Spaceballs
“Heeeeee is a pirate, a jar of whiskey and a bottle of winnnneeeeee”
) 
