Cybersecurity risk forecasting.

Ted Wagner: [Background Music] A risk model like a FAIR model with the added benefit of like quantification using revenue and vulnerability data enables business leaders, which is really the key, where security professionals can start speaking in the same language as business leaders, which is risk management. We can really have an impact on business decisions and how we can reduce risk within a business context.

Richard Fazzini: We're going to look back on the cybersecurity industry and say to ourselves, I can't believe we were deploying all this technology with no understanding of how it could effectively manage risk to a business.

Myrna Soto: Cyber risk has been a significant challenge for corporate boards because there is a lack of quantifiable metrics that can be equated to the concept of integrated risk management.

Bob Zukas: Once they have the ability to put this in economic terms, then it changes the game in terms of how they start to mitigate these risks. Not every firm is doing this. The vast majority are not yet, but the leaders are. They're understanding.

Kevin Richards: Without a financial representation of cyber, we're not doing our job.

Rick Howard: That was a collection of quotes taken from veteran cybersecurity thought leaders in order of appearance. Ted Wagner, our own CyberWire hash table subject matter expert. And Richard Fazzini [assumed spelling], Myrna Soto, Bob Zukas, and Kevin Richards, all taken from the Cyber Risk Solutions website earlier this year.

Unidentified Person: Oh, yeah.

Rick Howard: And fans of this show know that I have been trying to get my hands around how to calculate cyber risk for over a decade now. I've read all the best books on the subject. "Superforecasting: The Art and Science of Prediction" by Tetlock and Gardner, "How to Measure Anything in Cybersecurity Risk" by Hubbard and Seiersen and "Measuring and Managing Information Risk: A FAIR Approach" by Freund and Jones, all Cybersecurity Canon Hall of Fame Inductees. I've interviewed most of the authors for either the Cybersecurity Cannon Project or the CyberWire, and some of them are friends of mine, Richard Seiersen and Jack Freund. Richard and I even presented together on the subject at the RSA conference a few years back, and Jack reviewed the chapter on risk in my book, "Cybersecurity First Principles: A Reboot of Strategy and Tactics," available now on Amazon or wherever you get your books. And up to now, I felt like we were all just a bunch of rebels shouting into the wind and not gaining much traction. But I think that's starting to change. It feels like the infosec community is beginning to move in our direction. My indicator for this positive change is that I'm starting to see security vendors incorporate some of these ideas into their products. Specifically, I found two of them, Cyber Risk Solutions and ProcessUnity. So, hold on to your butts.

Unidentified Person: Hold on to your butts, butts, butts.

Rick Howard: I'm going to talk to these vendors to see what's driving the change.

My name is Rick Howard, and I'm broadcasting from N2K CyberWire's secret sanctum sanctuary studios, located underwater somewhere along the Patapsco River near Baltimore Harbor, Maryland in the good old US of A. And you're listening to CSO Perspectives, my podcast about the ideas, strategies, and technologies that senior security executives wrestle with on a daily basis. I talked to Fred Kneip early this summer, 2023. He is the founder and former CEO of a company called CyberGRX, a security vendor who uses many of the risk forecasting ideas that I've talked about on this show and explained in my book like super forecasting techniques, fermi estimates, and Bayes algorithm to help his customers assess third party risk. I said former CEO because just as we were doing this interview, Fred was putting the finishing touches on a merger with another company called ProcessUnity, where, as of this broadcast, Fred is the new president. The merger combines ProcessUnity's third-party risk management platform with CyberGRX's Global Risk Exchange. The problem that ProcessUnity solves for their customers is streamlining third party vendor risk assessment. For example, at N2K, we have hundreds of sponsors that buy ad packages for our shows like AWS, Rippling, and Expensify, just to name three. All of them are potential material cyber risk. How do I, as the CSO, evaluate each in turn in terms of material cyber risk to N2K? On the flip side, N2K sells subscription services in the form of CyberVista training packages and CyberWire Pro services to enterprise customers around the world. How do their CSOs evaluate if N2K is a material cyber risk to their organizations?

Fred Kneip: So CyberGRX is a -- it's a third-party cyber risk management platform kind of built on the concept of a one-to-many exchange. And what we're doing is we're recognizing that the companies are part of a growing ecosystem of vendors, suppliers they rely upon to deliver their core offering, and no one has the capacity to go out and evaluate the risk that exists across that whole ecosystem of vendors and suppliers. And historical approaches like sending out a bunch of questionnaires are just not scalable. Why don't we do that once in a high quality, thorough way? And then that data resides in our exchange. It can be shared or accessed multiple times. And so, that's the one-to-many exchange. GRX stands for Global Risk Exchange.

Rick Howard: So, give me some detail about how you do this. Like a company like, let's say the CyberWire.

Fred Kneip: Sure.

Rick Howard: Or they fill out the information and then what happens?

Fred Kneip: Yeah. So you think of it as there are two parties to any of these interactions. There's- Those who are consuming the data, typically the customer, and then those who are providing the data, those typically, the service provider of some kind. And interestingly, almost everyone out there is both. So it's a bidirectional concept in that sense. So take CyberWire, for example your customers will say, OK, I need to understand your cybersecurity posture. We're going to send you questionnaires. Or alternatively, if you're on the CyberGRX Exchange I can access that data. It's in a standard, structured format that I know how to read quickly and I can respond very quickly. And it actually accelerates your deal cycle, interestingly. And then you flip on the other side you have third parties you rely upon, the video conference service you're using today or others that, hey, wait a minute, I might be putting sensitive information. I should probably do some kind of risk assessment myself. And so, you can actually play on both sides of the exchange. And many of our largest customers are actually bidirectional.

Rick Howard: What you guys are doing there is interesting is it kind of fits in the middle of what I understand risk forecasting is. There's this concept of outside in analysis and inside out analysis. And outside in, you correct me if I'm wrong, is just what is the generic risk of doing something? Could be anything. And then inside out is all the things you're going to add more evidence based on how the individual company protects itself. But you guys are kind of in the middle there, right, that you're doing an outside in, but with some very specific data?

Fred Kneip: We're marrying up inside data that's been provided by our other customers who tell us how they use these third parties. And then we brought the outside of, OK, we have [inaudible]. But that's just the first step. That's the inherent risk.

Rick Howard: Yep.

Fred Kneip: That, I think the real magic or something we're really the most proud of is what we call our residual risk forecasting. And if you think about the same scalability, we go to the side where customer comes to us or go through the inherent risk mapping and say, great, I've got 500 critical people I want to focus on. Well, only 200 of them are on our exchange already. [Inaudible] it's the ones you'd know, AWS, Google, Azure or whatever it might be. However, they now have to go the process of the other 300 need to join our exchange to get all that data, well, that takes time. Sometimes it's easy, sometimes it's not.

Rick Howard: Yeah.

Fred Kneip: And- In that process. So in the interim, what we've done is we've said, OK, let's try and predict how those other 300 or whatever number of companies are going to respond to the standard CyberGRX questionnaire. And we're going to use our 14,000 completed assessments to build that prediction. In a very simple model what we do is we go and find a new company. you've given us, whatever, company x. We find all the externally visible data that we can. So, to your point on that externally visible, we will look at how many employees do they have, what's their revenue, what industry are they in, what region are they in, what does our partner risk recon say about them? What does our partner recorded futures say on the dark web? Do we see evidence of potential leakage, et cetera? We bring all that data and build what we call an external profile and we take that profile and we say, OK, let's go back to our 14,000 companies say, who else has a profile that looks as similar as possible to this? And let's build a cohort of maybe 100 to 200 companies that look as similar as possible. And how did they respond to each individual question on our questionnaire? And sometimes they all answer the same way. That gives us pretty high confidence that, OK, we're going to be able to predict that these guys are going to answer this way. And sometimes they're all over the place. We have no idea how they're going to answer it. And so, we'll have varying levels of confidence in our ability to forecast answers to every single question in a questionnaire. We've gotten now to up to 91% predictive accuracy. So that means, you know, on average, it's probably a little around 80% but we can get in certain sectors over 90%, which is pretty powerful. You know, you give me thousand third parties, I can give you with 90% or 90 plus percent confidence how they're all going to respond to a couple of hundred questions around their cybersecurity profile, that can be a very powerful tool. And --

Rick Howard: Well, the big epiphany I've had this last year is, you know, it used to be when I was trying to calculate risk, most security practitioners like us would have said, you know, we need all this precision. We need to know exactly what the number is. And what I've learned is that that is not the case at all. We just need a good enough answer.

Fred Kneip: Yeah.

Rick Howard: So, to your point, when you guys bring in a company, you're going to guess, and it's going to be pretty accurate most of the time, even if it's off a little bit, it's probably close enough to make the decisions that the security team is making about that company.

Fred Kneip: And your- that conflict is there, that precision versus speed concept. And everyone wants to say, I want every single detail updated daily on this company because that gives me as good as it gets. If you have 10,000 third parties and that's incredibly expensive impossible to deliver. Let's use the auto inherent risk to say, OK, let's take your 10,000 and focus it on the top 500. Then we can use our predictive risk profiles to help you identify here are the 10 that you might want to go and really dig in because they could really hurt you, and we're nervous about them. You know, risk management is not about risk elimination it's about focusing prioritizing where do I get the greatest risk reduction for dollar invested.

Rick Howard: Yeah.

Fred Kneip: And we help people streamline and focus their time and their energy to where we think the greatest reduction happened.

Rick Howard: You all know that I'm a giant fan of the MITRE ATTACK Framework. We've done several episodes of this podcast on it and it features prominently in the Intrusion Kill Chain Prevention Chapter, Chapter 4 in my book. My fantasy world is that I want the tools in my deployed security stack to forecast the likelihood that any of the 150 plus nation state attack sequences tracked in the attack wiki like ferocious kitten or mustang panda, or nomadic octopus are in my network. That should be simple enough, right? We know the attack sequences of each of these campaigns, the specific tactics, techniques, and procedures that each campaign uses across the intrusion kill chain. We should know the exact prevention and detection controls designed and deployed in the security stack to defeat each of the campaigns. If nomadic octopus has, say, 100 steps in its attack campaign and the security stack is alerting one of the steps, then it's likely not the hackers behind nomadic octopus. But if the security stack is alerting on 80 of the 100 nomadic octopus steps, then it's extremely likely you have the nomadic octopus hackers inside your network. Here's, Fred.

Fred Kneip: And one of the things that a standard data set enables us to do to answer your question even further is that instead of just saying, OK, here are the controls- here's what they have and they don't have, good luck. They have SSO and they have backups, good. What we've done is we've now taken the MITRE ATTACK Framework and we've mapped it to our question set and said, let's look and evaluate, you know, all these known attack paths. Do they have the controls in place that could have stopped each of those attacks? You know, we can have a kill chain analysis of here's how that attack happened. Nope, they could have stopped it right to the inception. This one would have gotten through, but this, you know, whatever segmentation of the network would have stopped them here, whatever it might be. And so, you can see and you can kind of map our controls to each of those different steps in a kill chain and identify where there is potential risk, and what are the most commonly, you know, exploited gaps, et cetera, in an industry so we can find, OK, here the controls they have or don't have, but here are the ones that really matter.

Rick Howard: I'm so happy that you're talking about the MITRE ATTACK Framework. Not enough of use that as a tool to defend our networks. Most of us are in the passive defense, kind of thing. You know, we're going to put controls in place for any kind of generic adversary, but hardly any of us -- And there's lots of reasons for it, but they don't put specific controls in for, let's say, Wicked Spider, because, I don't know, it's hard somehow. But what's clear to me is that we know what -- how Wicked Spider operates across the kill chain. And if we know that, why wouldn't you put those controls in place?

Fred Kneip: As new threats evolve, what we can do is we can then map them out, either MitreMap them and we'll just take that from them, or we'll do it ourselves. We have a team that focuses on that. And then you can now go back and look at your ecosystem. If that's 110, a hundred, a thousand third parties say, who of these is potentially susceptible to this attack? And let me index that to my inherent risk. Who do I care about and is susceptible to this attack? And so, one of the other major issues that occurs in third party risk management is that point in time concept. I did an assessment of them back in March. But I never thought about, you know, this type of attack. And now that's the whole- that's the number one ransomware attack that's happening. Well, our data is dynamic. It's on the exchange, it's constantly being updated by our passwords, and you can go back and it's comprehensive.

Rick Howard: Bayes algorithm is the underlying math theory that makes super forecasting techniques work, and it's designed to handle these dynamics. To put it into terms that CSOs can understand, you start out with a basic guess about what the probability of material impact is due to a cyber event in the near future. Let's say your first guess is 20%. And over time, you gather more and more evidence either about the outside security landscape, like a new log 4j type of attack in the wild or an improvement to your internal security stack, like maybe you just implemented single sign on for the entire organization. And then you adjust the initial forecast up or down depending on what happens. Bayes algorithm allows you to make continuous adjustments to your risk forecast as your specific situation changes.

Fred Kneip: And it goes even, maybe I get a little weedy on you here. But we all go back to the inherent risk questions of how do you use this company? If you use them, you need constant uptime. Then actually you're pretty worried about a DDoS attack because if that company goes down, you have a problem. So we'll actually overweight DDoS relevant controls if you say that's important to you. I don't care if they crash. It's a law firm. I don't care if their website crashes. But man, if that data gets out, it's a data loss. Now, I'm very focused, OK, we'll overweight those controls. And so, we actually prioritize the risks or the areas of concern based on how you tell us you use that third party. It's not just a generic ABC score. It's actually- this is how it scores for you and your use case.

Rick Howard: So let me double down on that. Your people that submit their info to your platform, they can say, you know, we've updated the Wicked Spider control set. Instead of 100 controls, we now have 150. And they can just automatically give it in. And so, your- when they try to interact with some customer down the line, they're getting the up-to-date control.

Fred Kneip: The more commonly someone is burying their CyberGRX assessment, the more incentive they have to keep it up to date and as fresh as possible. Because it's, you know, I take ADP as an example, they're sharing it every day. And so, they want that to be reflective of their security. And people typically are improving their security versus reducing it. So they want that- they want to keep that data fresh. The way the CyberGRX platform works is a third party. The provider of data can update their assessment at any time. I can finish and like a week after I did it the first time, I put in a whole new endpoint security program. I can log back in. I can update those controls, they'll revalidate. And now I present as such. And it's a new data set.

Kevin Richard: Kevin Richard, I'm the President of Cyber Risk Solutions.

Rick Howard: I met Kevin this past summer, 2023, at the ChiCyberCon conference in Chicago. He gave a talk on the current state of CRQ, Cyber Risk Quantification.