The Perfect Attack

Written November of 2019

DISCLAIMER: This is not your classic IT case about success and technical skills. Instead, we want to tell you a story containing anecdotes and observations from one of the most successful cyber-attacks in most recent history. We have cut out the journalistic middleman and the detached experts. The story is not told by those who were hurt, but by those who cleaned up and fought off the attack. Come with us behind the scenes, experience a war room, and hear how much pain it causes. Welcome to the story we all wish did not take place. The pictures in the story were not taken in connection with the actual company or attack. 

The External Threat

Cyber-attacks have become a part of our world and today, all companies and their IT departments are operating in a constant state of fear. In the most recent 4-5 years, the various IT media have placed a focus on stories about all possible forms of cyber-attacks. There has been no shortage of advice and guidance as to how to act. At edgemo, we also did not hold back. However, in the year 2019, we have to recognize that all parts of the business sector, in the East and the West, from smallest to largest and even the gigantic businesses still have deficiencies when it comes to preparedness and means in response to the threats out there. This is a sad fact and one we will return to later in the story. First, we must go fight the Russians and repair thousands of servers, manually.

”I Want My Datacenter Back”

Let’s turn back the clock to May. Or more specifically, to the Saturday before Pentecost, as Torben Christensen specifically delineates it. This might be because the following events turned, what was otherwise a well-deserved and planned public holiday and a while thereafter, completely upside down. “I got a call late in the afternoon. It was their Datacenter Manager who informed me that they had been taken down by ransomware and now he wanted his datacenter back. He sounded utterly exhausted.”

The customer who is a large global conglomerate had not only been taken down locally in Europe, from where he was calling, but across the entire world. More than 400 locations in more than 40 countries affecting more than 45,000 clients – everything. Torben explains: ”When he called, it had been a little more than a week since the attack itself. At first, they had tried a couple of very large foreign consultant companies who were closer to them, but to no avail. After that, they had fought on their own, but now they had given up. Wanting to handle things on your own is understandable, but it WAS not possible. Not when you are faced with what they were. It was an incredibly successful attack.” 

Code Red: Special Forces Move In

Torben explains that edgemo has not previously provided consulting services to this customer. “We had sold them a massive amount of hardware, but that was about to change quite substantially,” he says and continues: “I was not told a lot. Merely that I needed to gather a team. So, on a Saturday evening, I am calling around and at about midnight, we had the special forces ready. Early Sunday morning, we had a conference call on Skype. Everything was moving quickly and it was determined that we would send the first team to the location right away. As soon as Monday morning, we sent the second team.”

Infrastructure Specialist, Kåre Overgård recalls: ”When the phone rang, I was at a Phil Collins concert and the first thing I said to Torben was that I did not feel like I knew a lot about ransomware. But Torben was persistent and said; We need to send people in who know a lot. We need to go deep and wide and we need to come at this with a full deck of expertise. We do not know what they need yet.”

”We all had very different backgrounds so it was easy to distribute the tasks among us and they were very impressed by that. We just got to work. We delivered what they needed, quickly. We made the decisions ourselves and we stood by them. That is what is most important to a company in that situation since they do not have time to hand-hold.”

Out of This World

The character, extent, and success of the attack was solidified by what the edgemo people were met with. “What we were brought into was out of this world,” Torben says. “It was complete chaos. It looked like trenches in a war, which it actually was. People had not slept at home for a while. There were people in sleeping bags and it smelled of sweat and stressed-out people. People were zombies, but happy to see us,” Infrastructure Specialist, Alex Sørensen, who was part of the first wave, explains. “Normally, the large domicile is a place for guards, parking attendants, and what not. Now, the parking attendants had been sent home in order for help to get access.” 

Alex explains that the domicile’s large cafeteria was closed. “The local Asian grill was making a killing and the office was flooded with carry-out bins. It was extensive. Luckily, the cafeteria opened after some time,” he says and shares a story which illustrates the desperate need for help. “We were on our way to lunch and suddenly, we saw a crew of six men from Portugal making their way across the parking lot with suitcases and all. They were headed to the plane. They were VERY quickly chased down and stopped. They were also there after lunch and for a good while thereafter.”  

Serious Business

The responsible Russian “militia” meant serious business. The attack was so successful that it would send chills up any IT Manager’s spine. Being a former IT Manager, this was something Torben could relate to: “The ENTIRE conglomerate was shut down. The backup servers were encrypted to a point where a data restore could not be run and they had no Active Directory,” he says. “No AD was running anywhere in the world and just in the datacenter we were in, more than 7,000 servers needed to be recreated manually. Even though 80% were virtual, there were no backups so everything had to be manually installed.”

Alex elaborates: ”There was not a backup of all servers, but quite a few were backed up. Unfortunately, on many systems, they only backed up the data and not the operating system and the application which meant that everything had to be installed manually. I think many were surprised to find systems on which backup was actively deselected. That was surely an expensive lesson learned.” Kåre adds to this: ”Everything had been turned off so one by one, we needed to turn on the servers in offline state to check if they had been infected and then, they needed to be rebuilt. It was a massive undertaking.” And that was only the datacenter which edgemo was located in. On another continent was a datacenter of similar size. All in all, this involved more than 40 countries in which not one singular PC was operational. More than 45,000 people were out of work across the world. Fortunes flew out the window every minute, for weeks.

Torben has the following to say about the spirits in the company: “Everyone was affected. The work had come to a complete stop. General panic spreads and the support system quickly falls apart. This all equates to success for the hackers. It is virtually impossible to run manual processes – EVERYTHING in this large corporation is run by IT. The only thing they could do was open the gates and receive products. As such, they were able to run a few various processes manually to a very limited extent,” he says. “The immediate result was that a vast amount of employees were sent home for a long time. Many thousands of people and for many weeks. That truly hurts the economy – also for those at home because they were unable to process payroll.”

”When you then stop to realize that they could have minimized the extent quite extensively, it is almost unbearable…”

State of Emergency – in the War Room with edgemo

Going back to the location and May of 2019. ”We dove into a massive infrastructure which we knew nothing about. From there, we all took on tasks thrown at us and we did not ask any questions,” Torben says, and Alex adds to this: “As edgemo specialists, we are used to being in control of things. We control the progress every day. However, here, we were initially required to act based on their decisions, which was a new experience. Yet, we were given many tasks and we quickly realized that they had nobody coordinating the efforts. Nobody shared anything. Therefore, we started to share things among ourselves and more importantly, we started to document the processes. Nobody at the location had a tool, so we built one. The customer took notice of this and so did the many other external individuals called in to help. They started looking over our shoulders and used our tool and our documentation. That way, they did not have to start from scratch each time and six people did not have to do the same thing. We were able to gain control over a lot of things.”

Torben adds to this: ”It was critical to us and the assisting forces from Portugal, Germany, and other countries, that people speed things up. Nerds truly flourish when they can be a key part of a war room. It is an incredibly inspiring experience.”

When Everything is Critical

Before we can finally let you know how the damage could have been limited, we have one more notation. How does one measure an attack? Pure violence, finesse, or strategy? Our counterpart in this matter clearly did not play Battleships as children, but they probably beat up a bunch of kids on the playground so we will withhold any praise. Regardless of method, in order to understand the enormous success of the attack, one must look at the corporation from the outside. Look at the connections because they outline the complete extent.

What we are dealing with here is a conglomeration which is part of a global mega sector. A global critical key player with especially time-sensitive deliveries in supply chains within this mega sector, which again consists of several inter-connected business and societal sectors. If one player is down for the count, it affects everyone because human lives are on the line. As such, the attack was about more than “just” stressed out employees and a massive dark spot in the annual accounting, it reached further out into the world.

Because of this, it should be written in large invisible letters: SECURITY IS EVERYTHING. Unfortunately, reality was very different from that. This also pointed out the enormous paradox that the cause of the extent of the attack, unfortunately, lay with the customer itself.

- The bullies did not deserve the success they achieved at all.

”An Extreme No-Brainer!”

Even though this is all about a well-run organization in a just as well-run global conglomerate, there were still gaps in the security. According to our three musketeers, this was an issue of classic pitfalls which many, far too many, fall into. This is also the reason why this story is being told.

Torben gets right to the point: “Let me start by making this clear; One checkmark could have avoided 80-90% of the extent of this attack,” he says. “One has to ask oneself: Did we activate LAPS (Local Administrator Protection Solution)? With that, all local administrator passwords are randomized, and it really just requires one checkmark to activate. This is an extreme no-brainer. It is such a no-brainer that we actually do not even ask people if they have LAPS now a days – it is the equivalent of asking if you even have passwords.” Torben adds that if you have the same administrator passwords on all machines in your organization, it is the equivalent of not even having any passwords at all. This was the case in this instance.

Kåre chimes in: ”They actually had some expensive and impressive security measures in the upper levels in the system, however, since the basic part was lacking, it really did not make a difference. Even though they were in place, this ransomware just flew from the bottom up through it all and lit fires along the way. Basics are key when designing your security setup,” Kåre says and explains that they assisted an employee who had not changed passwords in 12 years.

”This attack began on the inside, but I have to add that the built-in Windows Firewall and UAC were also disabled. Despite it being a bit troublesome to work with, it is there for a reason.”

The Scandinavian Rescue

But then what? Did they make it? Naturally, the story ends with the attack being fought off, but it did take a couple of months.

The happy ending came about in that the previously mentioned very intelligent Datacenter Manager played a very key role. He happened to have a domain controller located somewhere in Scandinavia, completely offline. “From here, they could pull their entire AD schedule. Step one was to recreate their AD. They managed to do so and from there, it was just hard work,” Torben says. At one point, they found a remnant of the code in the system which affected approximately 2000 re-created servers and that sent all those back to square one. It was truly a race against time. Ultimately: If we had not had this Datacenter Manager, we would instead have spent our time recreating the company’s IT from the ground up,” he says.

Take Control of your Processes – and Your Backup

Organizations, big and small, need preparedness to include procedures for both daily operations and crisis situations – as Alex says it. These days, you need to be ready to act. ”They were not at all ready to rebuild things to this extent. To their credit, they actually had a “factory” in which to build the servers. But when they suddenly needed to build such a large amount, it could not meet the requirement. It had a bunch of manual steps which is fine when you are building a few per week, but in this instance, we needed to build a couple of thousands per week”.

Kåre adds to this: ”When it comes to backup, it is also very important to have your policies and processes in place. The way they previously ran backups was taking a snapshot once every twenty-four hours. When you are subject to an attack like this, you have to look at what is worthwhile. The hours just kept flying out the window,” he says. They had started with rolling back to an image which was a week old. My guess is that they could have saved thousands of consultant hours by going back further, for instance a mere eight days when the machines were not infected. Then, they could have built the servers and machines in bulk and saved days of work and millions of Danish kroner.”

You Need to Learn from Your Mistakes

Kåre talks about the time after the attack. “When the storm had calmed, we visited the customer again to help run through new passwords. Even that process was truly chaotic. They made very rash decisions. They told 45,000 users to change passwords giving them two days’ notice. It is not necessary to stress that it was chaos and they also did not forewarn us. In fact, they repeated this successful method later on in connection with a large update. Yet this just proves how difficult it is to break habits and systems. And this company is far from alone. Unfortunately, we see this far too often that people think that all is in order when they buy some tools and systems, but the systems cannot handle it all on their own.”

Alex adds to this: ”It is the be all, end all, to create a preparedness to act. You cannot be caught off guard. The systems for backup and updates have to be in place, but preparedness and being ready is just as important. If not, an ordinary restore task which would normally take 40-45 minutes ends up taking 6-7 hours as was the case here. You have to be ready to recreate EVERYTHING at once. Quickly and with agility.”

Thus ends the tale from the backside of one of the large attacks. We have told you what we are permitted to and what we can. The customer knows we are sharing the story. However, the anonymization plays a key role when this is about all of us getting smarter. The examples could be from anywhere. The pitfalls are well-known and the situations represent what could happen. Everyone knows the feeling of being sent back to square one, or even further back than that.