SolarWinds through a first principle lens.

For the past year, I’ve been developing a set of cybersecurity strategy theories based on first principles. The key word in that description is “theories.” After years of running around in the industry working as a security executive or advising the same, I have seen many things tried. Some have worked, and some have failed. But that experience has allowed me to build this “grand unifying theory” about how to do cybersecurity in the future. The question that I’ve been asking myself this past year is, would it work? Would that set of theories prevent material impact in the real world? Would they have worked against the latest SolarWinds attack?

The setup: preventing material impact from a major cyber incident (like Solorigate).

Let’s just say I’m the CSO of a medium-to large-sized organization. With a nod towards the “Iron Man” and “Avengers” movies, let’s call it Star Enterprises. For the past year, we’ve pursued such goals as resilience, intrusion kill chain prevention, zero trust, and risk forecasting. Although many more things remain to be done tactically, the Stark Enterprises executive team, Tony and Pepper Pots, have been pleased with the overall progress. For the sake of this exercise, assume we’ve deployed at least 50% for all four strategies. And then the SolarWinds incident hits us. Would those deployed first principle strategies have prevented a material impact to Stark Enterprises? Let’s find out.

Attribution of the SolarWinds supply chain compromise.

The event has collected a growing list of campaign and adversary group names from various commercial security vendors: 

• Dark Halo named by Volexity
• SolarStorm named by Unit 42
• Solorigate named by Microsoft
• UNC2452 named by FireEye

For the purposes of this essay, I will use UNC2452 to refer to the adversary group and SolarStorm to refer to the campaign.

Unfortunately, there have also been unsubstantiated preliminary reports attributing the attacks to APT29:

• AKA Cozy Bear
• AKA the Russian Foreign Intelligence Service, or SVR
• AKA YTTRIUM

On 5 January, the FBI, the NSA, CISA and the ODNI released a joint statement blaming Russia.

The fact is that the SolarStorm attack sequence matches none of those other known campaigns. As far as I can tell, the only previously known procedure deployed by UNC2452 was the use of a piece of malware called Cobalt Strike. According to Malpedia, “Cobalt Strike is a paid penetration testing product that allows an attacker to deploy an agent named 'Beacon' on the victim machine.” Over the years, Cobalt Strike has been used by adversary groups from Russia to Vietnam, China, and Iran. 

According to Recorded Future’s John Wetzel, U.S. Government officials will release technical evidence soon that will point to Russia. They haven’t yet for fear of revealing sources and methods. I understand about protecting sources and methods, but until somebody steps up and makes the evidence public, the attacker could be anybody at this point.

By the way, there is a way to do this without compromising your sources and the U.S. government has done it many times in the past. They pass the technical evidence to the commercial security vendors, specifically, to the Cyber Threat Alliance. This group of roughly 30 security vendors with some of the best intelligence teams in the world will confirm the intelligence in their own product lines and make attribution assessments while they're doing it. This leaves the government out of it and does not reveal their sources and methods. The U.S. Government hasn’t done that yet and that makes me suspicious of the validity of the Russian attribution claim. 

Don’t get me wrong. It’s probably the Russians. There are only a handful of nation states that have this kind of capability and it feels like something the Russians have done in the past like Sandworm. But so has China. We are just not there yet in terms of actual evidence to prove the assertion.

Unknown unknowns: the difficulty of preparing for a zero-day campaign.

The bottom line is that UNC2452 built the campaign almost entirely from scratch, a zero-day campaign if you will. This is highly unusual. For the past ten years, most adversary groups, even the stealthy ones, haven’t created brand-new attack campaigns with original components for each phase of the intrusion kill chain. They haven’t had to. Usually, they change the section of the attack sequence that doesn’t work anymore because victims have figured out how to block it, or the adversary group has innovated some better way to do it. To invent entirely new attack campaigns is expensive and time consuming. In that way, SolarStorm is on the same level of complexity and resource commitment as another famous zero-day campaign: the 2010 Stuxnet campaign. That cyber operation penetrated the Iranian nuclear plant in Natanz for the purpose of degrading the production of nuclear bombs.

By creating a zero-day campaign, UNC2452 nullified a cornerstone pillar in our first principle cybersecurity strategy wall: intrusion kill chain prevention. That strategy leverages the observation that cyber adversaries don’t typically do this, and it shifts the infosec teams away from blocking technical artifacts in isolation, like malware or zero-day exploits, with no context about what the adversary is trying to accomplish. The idea is that you deploy prevention and detection controls for every tactic, technique, and procedure across the intrusion kill chain for all known campaigns. In that way, even if the adversary changes a specific TTP of the attack sequence and makes it invisible to threat hunters, the other prevention and detection controls still in place will prevent the success of the operation. Ryan Olson, my friend and colleague from Palo Alto Networks Unit 42, and I wrote an entire paper on this concept that we published in the Cyber Defense Review this past fall.

With a zero-day campaign like SolarStorm, that strategy doesn’t work. Stark Enterprises doesn’t have any detection or prevention controls in place to stop the success of the operation because every step in the attack sequence is unknown except for the Cobalt Strike malware. That knowledge might be enough to stop this campaign, but probably not. We have to admit it: at the time of the SolarStorm attack, our intrusion kill chain prevention strategy most likely didn’t prevent material impact to Stark Enterprises. 

This leaves us with the other three first principle strategies: zero trust, resilience, and risk assessment. Would some combination of those have prevented a material breach? Before we can answer that, we have to understand how the SolarStorm attack sequence worked – at least the part we know about it.

Details of the SolarStorm campaign.

In terms of the intrusion kill chain model, we know that at least one initial entry vector was a backdoor trojan inserted into the SolarWinds Orion software platform and automatically delivered to roughly 18,000 SolarWinds customers as a software update. According to the SolarWinds website, Orion is an orchestration platform designed to manage the entire IT stack, from network performance to configuration to IP management and so on. It’s similar to security platforms like those offered by Fortinet, Check Point, and Palo Alto Networks; but in the case of SolarWinds Orion, it’s only for IT orchestration.

UNC2452 used a supply-chain-attack vector as the initial delivery mechanism. By supply chain, we mean that it didn’t attack its victims directly. Instead, UNC2452 attacked a key software component, in this case Orion, that potential victims were likely to use. UNC2452 inserted some backdoor code into Orion, rode into the victim’s network via the legitimate software update channel, and then used the backdoor to compromise its intended victims. 

There have been several supply chain attacks over the years, but arguably the two best known are: 

• 2013: Target supply chain attack: Fazio Mechanical Services, Target’s heating, ventilation, and air conditioning (HVAC) supplier. Purpose: cybercrime.
• 2017: Ukraine (NotPetya) supply chain attack: MeDoc, financial software package used by many eastern European companies. Purpose: the Russians conducting continuous low-level cyber conflict and influence operations against Ukraine.

The second thing we know is that UNC2452 leveraged their victims’ identity management systems. They managed to steal the secret key from an on-prem single sign-on (SSO) management server responsible for federated accounts. In the case where the victim was a Microsoft customer, UNC2452 leveraged the Active Directory Federation Services (AD FS) server. 

If you remember from S2E7 and S2E8, we talked about identity management and the federated model. We learned that you get this kind of transitive property of trust with federation. If the University of Michigan trusts Ohio State University, and Ohio State University trusts its CISO, Helen Patton, one of our Hash Table subject matter experts, then the University of Michigan trusts Helen. In other words, they believe it is her, but they still authorize her access to U of M’s resources locally. She doesn’t get carte blanche access just because she’s Helen. U of M decides that.

UNC2452, through some process of Microsoft Active Directory privilege escalation, stole the private key used to sign Security Assertion Markup Language (SAML) tokens. Again from S2E7 and S2E8, SAML refers to a heavyweight XML variant language that facilitates one computer to perform both authentication and authorization on behalf of other computers. UNC2452 then used that secret key to forge trusted authentication tokens for cloud resources. In other words, they stole the secret key from an on-prem server to create authorization tokens for virtual systems in the cloud. 

This SAML technique, called Golden SAML, isn’t new. The NSA published an advisory on it back in 2017, noting that it has been used by both nation-state groups and other types of threat actors.

Zero trust and risk assessment would probably work but ...

Zero trust is the obvious strategy to leverage here. UNC2452 went after two systems that control key components of the Stark Enterprises network undercarriage: network management with the Orion orchestration platform and the identity management system with Active Directory Federation Services. Since Stark Enterprises is at least 50% complete on deploying the four pillar first principle strategies, it is likely that the risk assessment strategy identified these two systems as warranting prioritization and special attention. 

With the zero trust strategy, we are trying to reduce the Stark Enterprises attack surface by limiting employee access to only the resources they need to do their job and nothing more. It seems reasonable then that the Stark Enterprises infosec team would have greatly limited the number of employees who have access to those two systems. They might even have restricted the accounts further to people authorized to commit changes. Additionally, they would have insisted that the accounts those employees used to do those jobs weren’t the accounts they use when they’re doing their daily routines, like reading email and surfing the web. Lastly, the Stark Enterprises infosec team would have probably established special monitoring for two processes:

• Changes to network configuration 
• Creation of authorization tokens

The monitoring process would ensure that somebody else knew about those operations and could verify that the task was done with the correct procedures. Think of it like the two-person control protocols used by the Air Force and others to control the launching of nuclear missiles. The Stark Enterprises infosec team would deploy all of these zero trust tweaks in an effort to reduce the probability of privilege escalation. That would have likely stopped the SolarStorm campaign.

On the other hand, I realize that this zero trust plan has a lot of bureaucratic pieces to it. When the boss is breathing down your neck to get something done, most of us don’t have time for that kind of nonsense. Before SolarStorm, as the CSO of Stark Enterprises, I might have assessed the probability of a zero-day campaign impacting the business as being pretty small. Supply chain attacks had happened before, for sure, but they weren’t common. And with my intrusion-kill-chain prevention strategy deployed at 50%, I might have been feeling pretty good about my chances.

The epiphany for me, though, is remembering our very first cybersecurity first principle, the overriding atomic element that drives the four pillar strategies. As the Stark Enterprises CSO, I am trying to reduce the probability of material impact due to a cyber event. For any adversary trying to cause Stark Enterprises harm, the “crown jewels,” so to speak, include the identity system. If I don’t lock that down, I open Stark Enterprises up to all kinds of trouble. By reducing the attack surface of network configuration changes and authorization token creation, I might prophylactically prevent many future known and unknown campaigns from being successful. In that way, establishing these bureaucratic zero-trust procedures might be the very first thing I do. 

Resilience in the face of a major, sophisticated cyber campaign.

FireEye was the first commercial company that publicly admitted being a victim of the SolarStorm campaign. In season two, episodes three and four, we talked about incident response and the executive decision about when to go public with breach information. There are basically two choices:

• Announce early to get in front of the story, but risk public disfavor when additional information comes out later that might contradict your initial assessment.
• Announce later when the information is almost perfect, but risk public disfavor by being accused of withholding important information.

The FireEye CEO, Kevin Mandia, decided to announce early. On 8 December, he told the world, and the company’s stock price dropped from $15.51 to $13.54, not a huge plunge, but still significant. But because of the way Mandia handled the announcement with complete transparency about how the company discovered the attack, how UNC2452 conducted their attack campaign against FireEye, and the indicators of compromise that others could use to hunt for UNC2452 in their own environments, the stock dip was a minor setback. Just before Christmas, the stock price had rebounded to $24.84. 

There are other metrics that we might use to measure material impact that aren’t public right now, like: 

• Customer retention down the road
• Hitting their quarterly numbers at the next analyst call

For now, the way FireEye handled the announcement has been favorably received by the investment community, and the SolarStorm campaign hasn’t had a material impact on the business. By executing a well-thought-out incident response plan, Kevin Mandia and his team made FireEye more resilient.

Microsoft followed the same playbook and got similar results. Their stock dipped from $216.65 to $210.68 after their announcement, but had rebounded to $225.86 just after Christmas.

It’s interesting to note that there are potentially 18,000 other organizations that have decided to delay their breach announcement including SolarWinds. The Solarwinds stock price dropped from $23.56 to $14.50 right after the new year and shows no signs of rising. Of the eleven commercial firms we know were touched by the campaign, only FireEye and Microsoft made their public announcements early. Down the road, we will see which strategy worked for these SolarStorm victims.

Last words on first principles.

I think it will be years before we know the full extent of the attack sequence executed by UNC2452. We know some of the sequence details for sure mostly because they are so unusual. We by no means know the entire story of how the adversary group penetrated potentially 18,000 victims, though, nor what UNC2452 exfiltrated if they were successful.

Going through this Stark Enterprises exercise, however, proved to me that our four pillar cybersecurity first principle strategies are fundamentally sound. It also proved to me that you can’t focus on one and not the other three. One, two, or even three are unlikely to be sufficient. You have to do all four.

Reading list

S1E6: 11 MAY: Cybersecurity first principles.

S1E7: 18 MAY: Cybersecurity first principles: zero trust.

S1E8: 26 MAY: Cybersecurity first principles: intrusion kill chains.

S1E9: 01 JUN: Cybersecurity first principles: resilience.

S1E11: 15 JUN: Cybersecurity first principles: risk assessment.

S2E3: 03 AUG: Incident response: a first principle idea.

S2E4: 10 AUG: Incident response: around the Hash Table. 

S2E7: 31 AUG: Identity Management: a first principle idea.

S2E8: 07 SEP: Identity Management: around the Hash Table.

“A BRIEF HISTORY OF SUPPLY CHAIN ATTACKS,” by Secarma, 1 September 2018.

“Analyzing Solorigate, the compromised DLL file that started a sophisticated cyberattack, and how Microsoft Defender helps protect customers,” by 365 Defender Research Team and the Threat Intelligence Center (MSTIC), Microsoft, 18 December 2020.

“A Timeline Perspective of the SolarStorm Supply Chain Attack,” by Unit 42, Palo Alto Networks, 23 December 2020.

“Cobalt Strike,” by Malpedia.

“Countdown to Zero Day: Stuxnet and the Launch of the World's First Digital Weapon,” by Kim Zetter, published by Crown, 3 June 2014.

“Cybersecurity Canon,” by Ohio State University.

“FireEye shares jump back to pre-hack levels,” Melissa Lee, CNBC, 23 December 2020.

"Implementing Intrusion Kill Chain Strategies by Creating Defensive Campaign Adversary Playbooks," by Rick Howard and Ryan Olson, The Cyber Defense Review, Fall 2020.

“Orion Platform,” by SolarWinds.

“Sandworm: A New Era of Cyberwar and the Hunt for the Kremlin's Most Dangerous Hackers,” by Andy Greenberg, published by Doubleday, 7 May 2019. 

“SolarWinds Attribution: Are We Getting Ahead of Ourselves?” by JOHN WETZEL, Recorded Future, 30 DECEMBER 2020.

“SolarWinds Campaign Focuses Attention on 'Golden SAML' Attack Vector,” by Jai Vijayan, Dark Reading, 22 December 2020.

“SolarWinds hack officially blamed on Russia: What you need to know,” by Laura Hautala, Cnet, 5 January 2021.

“Solarstorm,” by Unit 42, Palo Alto Networks, 23 December 2020.

“The Cybersecurity Canon: Countdown to Zero Day: Stuxnet and the Launch of the World’s First Digital Weapon,” by Rick Howard, The Cybersecurity Canon Project, 28 January 2015.

“Using Microsoft 365 Defender to protect against Solorigate,” by the Microsoft 365 Defender Team, 28 December 2020.