What does hospital downtime teach us about building AI-native organizations?

Zach Evans: Can't put the genie back in the bottle. I don't think we should put the genie back in the bottle. But our security practices and the way that we go about building solutions is going to have to change a little bit to account for the new risks that are being introduced because of this technology. [ Music ]

Johnny Hand: Welcome to AI Security Brief, where we're unpacking emerging AI threats, vulnerability research, and the strategic decisions that security leaders are making right now. I'm Johnny Hand.

Dustin Childs: And I'm Dustin Childs. Today's guest is Zach Evans, CTO of Xsolis, a healthcare data company that's been building AI-driven predictive models for over a decade, long before AI native became just a marketing term. Zach's team has produced more than 6.5 billion clinical predictions from a single model, and the company was built from day one to eliminate data silos between hospitals and insurers. The name of the company is literally silos spelled backwards.

Johnny Hand: Yeah, it's a great conversation. And we really started diving into the term AI native and what it actually means and how it's not just bolting on AI into your organization. Zack has a simple test that we talk about where we say, essentially, if you strip the AI out of the product, does it stop working? And if it does, it's probably not AI native. Then we go deeper into agentic AI, especially in those high-stakes healthcare environments, and how you need to onboard your agents the way that you onboard a new hire in your company with least privilege access. We also talk about why that Hugging Face and OpenAI incident was such a big wake up call for all of us.

Dustin Childs: If you're running agents anywhere near a highly regulated, high-stakes environment, be it financial, healthcare, critical infrastructure, this conversation is about what changes the moment an agent gets it wrong. And Zach didn't dodge the harder question either. What happens to your team's skills when the agent is the one doing the work? Let's get into it.

Johnny Hand: So I have to ask because I'm sure our listeners, I know I have, have been inundated with this term, especially a marketing term that we hear all the time, which is "AI native." We're hearing that being thrown around. So when you say Xsolis is an AI native company, what does that mean for your company, the way it's actually built? And then how are most companies using that term that's maybe different from your company?

Zach Evans: No, I think it's a great question. And, you know, it's really, when I think about what it means to be AI native, it's really about kind of, if you're AI native, it's not just a capability that you've kind of bolted onto an existing process. It's really something that has to be core to what you are building and what you have built and what you're going to build in the future. And so for Xsolis, what that means is -- you know, we launched our first predictive model driven by artificial intelligence over a decade ago. It's produced over 6.5 billion predictions on behalf of our clients. And so we've really built our entire infrastructure, our entire data model, really the way that we go to market around the idea of we want everything that we bring to market, we want to be touched and driven by artificial intelligence. Now, there's all sorts of different applications of artificial intelligence. You heard me talk about machine learning and some of those early applications of artificial intelligence all the way through generative AI and now into agentic AI. But when I think about being AI native, again, it's really about, it's not just a bolt on, it's something that really permeates everything that you do as a business and the way that you build everything for your business.

Johnny Hand: Now, the name, I happened to notice, I'm sure this was intentional, Xsolis is "silos" backwards. Was breaking down data silos between hospitals and insurers the founding thesis, or did that become clear over time?

Zach Evans: No, it absolutely was part of our founding thesis from the beginning. So you're right, the name Xsolis is the word "silos" spelled backward with an X put in front of it, because that's exactly what we wanted to do. As an organization, one of our core tenets is we really believe in data symmetry. You know, too often, within any number of relationships, not just within healthcare, but within industries at large, right, where you have, you know, two components of a marketplace, that it's not that they're at odds to one another, but that they have slightly different motivations, right? In the healthcare industry, where you have hospitals that are providing care and deserve to be compensated for that care, but that care is being paid for by a third party, an insurance company that has its own motivations. Well, what that type of relationship dynamic can lead to is kind of information hoarding, right? You know, one party or the other, you know, builds or receives or somehow develops data or information that gives them kind of, you know, the leg up on the other party within that marketplace. And we just thought that was a really kind of not the best way to think about that relationship. Because at times, it can actually lead to a bad outcome for the ultimate customer, which is the patient that's being treated.

Dustin Childs: The reality is is, you know, machine learning and AI workflows have been in place for many, many years. Even at TrendAI, we've been using them as a part of our detection models for many years. So it's not just starting, but we have seen a massive transition. So what does that look like for you guys? Are you seeing that score now be generated using AI or agents in a way that's different than when you started?

Zach Evans: Yeah, no, it's a great question. So the way we're thinking about, especially with the emergence of new applications for artificial intelligence, specifically agentic AI, really where we're seeing that happen is, you know, will this care level score ever be, you know, computed by an agent? Maybe. You know, one of the use cases that I could see there is, I mentioned, you know, we take in that medical record one message at a time and we're kind of constantly updating those predictions. An interesting use case for an agent would be an agent that decides, okay, what's a material bit of clinical information where we need to update that prediction versus, okay, that's an immaterial update to the clinical model and we don't need to update it. Like that's a really interesting use case for an agent. Where we think you can drive real value with AI agents is what you do with that data. So, you know, today, we update our model, we make a prediction, you know, we fire off a notification, or we assign a workflow task to a person in the hospital to respond to. And that works really well. But one of the problems is is that all of these hospitals are under resource constraints. They can't hire, they literally cannot find enough qualified people to do these jobs. So you end up with high stress, high turnover. You have too few people doing too much work. And so where we do see a lot of use cases for agentic AI is by automating certain components of what comes next.

Dustin Childs: That's interesting. You know, a lot of teams still treat agents like task rabbits, which is just smarter automation. Where's the line between automation, orchestration, and general agentic behavior, and why does getting that wrong matter for security?

Zach Evans: You know, the world really woke up to a new reality with OpenAI's hacking of Hugging Face, right? And it really laid bare a lot of the challenges with AI agents, right? You know, when you are just trying to automate a discrete task, you know, moving the box from point A to point B, that's a great case for automation, and you may have a really small agent that does that. You know, similarly with orchestration, right? You know, you build these really small kind of one-off purpose-built agents that then get orchestrated through these series of steps. You know, there's absolutely security concerns around that. But when you have these really tightly bound workloads, you know, you can define that blast radius, that risk, a little bit easier. When you get into full autonomous thought, right, of where, you know, you've given an agent a goal and you're saying, I want you to find the best path forward to accomplish that goal, you really have to design security from the very beginning and from the ground up. You know, I tell people quite often, I say, look, I say, when people are first starting out, even with generative AI, right, you know, you're writing your first prompt into Claude. Typically where you start is you're saying, well, I want to do X, Y, and Z. And you're telling that agent what you want it to do, but there's a gigantic blind spot in you're not telling the agent exactly what they are not allowed to do. And so those guardrails and those constraints, as you kind of get more mature and more experience in prompt engineering, you start learning to kind of define that negative space quite a bit more. Well, when you're dealing with agents, the risk and the need for that type of thoughtful security around both what you want the agent to accomplish but also explicitly what you don't want it to accomplish becomes that much more critical. And again, the ChatGPT hack of Hugging Face is a great example of that, right? The agent decided the next best course of action was that it needed access to the public internet and it figured out that Hugging Face had the answer it needed. You know, how you design around that from a security perspective is critical to ensure that the agent isn't able to make that choice. Even though, you know, in a vacuum, that is the next best choice, it doesn't understand the consequences of the choices it's about to make. And you have to ensure that you're building proactive security around that.

Dustin Childs: So I also see MCP servers as a real attack surface that most security teams haven't caught up with. What does a well-guardrailed agent connecting to a poorly secured MCP server look like in practice?

Zach Evans: You know, it's, man, it is a question I wrestle with every day. Because, you know, MCP is, it's a great, it's a great construct, it's a great idea of how to have these agents be able to reach into and access, you know, other tools and solutions in a seamless manner. I heard it described this way though, you know, what we really need to have happen to MCP servers is that, you know, MCP servers and like something like an API gateway get together and have a baby, where you have, you know, these very kind of fine-grained permissions and access controls that are role-based and are credentialed. And that way, when an agent reaches into an MCP server, it's not just the wild, wild west of everything that that MCP server has access to, but you have this really fine-grained credentialed control with really strong guardrails about what that agent is able to do when it goes through the MCP server. I think that's absolutely where we need to get. And then the other thing that you have to focus on is you have to treat an MCP server just like a new vendor that you're reviewing. You've got to do basically a complete security review to understand, you know, what the agents can do as they access that MCP server. You know, how is the data that you're sending through the MCP server, how is it treated? Is it controlled? What type of role-based access exists. You know, you have to treat it just like any other software vendor to understand the risk that you're trying to manage. Because the promise of the MCPs and the connectivity that they promise is absolutely crucial to the growth of agentic AI, but it is also oftentimes an unmanaged risk that can really present challenges to security professionals.

Johnny Hand: I want to shift gears a little bit backwards to, you know, a lot of times when you look at industries like healthcare, you look at industries like, you know, with OT industries and critical infrastructure and these things where there's a real impact when mistakes happen. How are you balancing the risk while you're building these agents and like, what does that redundancy and backup look like for you?

Zach Evans: You know, it's -- I spent some time as the IT director inside of a hospital, okay, and so I know what a downtime event looks like inside of a hospital, when clinicians have to go back to paper because the electronic medical record has gone offline, for whatever reason it happens to be. And so I saw it on the front lines of where, you know, you're handing a physician or you're handing a nurse a paper chart to chart in. And the reality was there were some clinicians that, you know, just based on when they were trained, they had never written on a paper chart. And so you had to do, you know, oftentimes some crash course training on exactly how to fill out a paper chart. It wasn't because they weren't, you know, smart enough to do it. These were incredibly intelligent clinicians. They just had never worked in that type of an environment. And that's a very real risk. As we take increasing amounts of work out of the hands of individuals and put it into the hands of agents, you have to be able to plan for what happens when the agent isn't available. A system can't just go down because an agent is no longer automating a bit of functionality. So you have to have manual workarounds within a system to provide redundancy. That's the technical solution. But you also have to have a people solution. That even as you automate work, that there are still people that understand what the underlying process has to be in case and when that agent goes offline.

Dustin Childs: So that's very interesting, because I'm sure you're familiar with the TV show The Pit?

Zach Evans: I am.

Dustin Childs: In season two, something like what you described happened where they had to go back to pen and paper versus their computers. I'm just curious, what's your opinion? Did they get it right, or did they get it close?

Zach Evans: It was eerily close, okay? Yeah, the real world example that I lived through when I was the IT director and there was a downtime event -- now this is, this is more than a decade ago -- but it felt very much like that. Of, you know, all the way down to, you know, as part of our downtime procedures, you know, if you remember in The Pit, right, they pulled out, it was like this rolling cart that had these sheets on it that people could document on. You had to plan for, okay, look, it's great that you have downtime forms. But what if the network's down and all the printers are down and you can't print out new copies? And so, you know, you had to think about things like that and be prepared for, you know, how you would continue to provide care. So yeah, The Pit, I will say, having lived through that, it was -- my heart started racing a little bit when we watched that episode because I had been there, and they're not wrong with how they depicted it.

Johnny Hand: We've had a really great conversation so far and what I wanted to do is, based on what we've talked about today and kind of our highlights, is there one thing that you want our audience to think about or maybe do when they hear this conversation?

Zach Evans: Yeah, I think the biggest thing I want the audience to think about is really how you bring in those security by design principles from the very beginning, right, especially with agents. This is no longer, okay, you know, traditional software engineering where maybe, you know, an application engineering team goes out and builds something and then there's a security gate somewhere before it goes into production. That's just not going to be enough. Security is going to have to have a seat at the table from the very first design session and to really, again, apply some of those principles around zero trust and other types of security best practices from the very first time that agent is being designed so you can really control and manage the risk that's being created when you hand off decision-making to a piece of computer code. You know, we can't put the genie back in the bottle. I don't think we should put the genie back in the bottle. But our security practices and the way that we go about building solutions is going to have to change a little bit to account for the new risks that are being introduced because of this technology.

Dustin Childs: Yeah, well, Zach, thank you very much for talking with us today. It's been a great discussion and I really enjoyed it. And I hope you can come back again in the future at some point and give us an update on you and what Xsolis is doing. But until then, just thank you very much for being here.

Zach Evans: No, thank you so much for the opportunity. I really appreciate it. And yeah, I would love to join you all again at some point.

Johnny Hand: Awesome. Thank you.

Johnny Hand: Dustin, what a great episode with Zach. And he brings so many valuable points, especially from the healthcare industry and what he's doing with Agentic AI. I think the one thing that we can really embrace is that you have to treat every AI agent like it's a new hire, right? You want to give least privilege access on day one, and then you only give it broader permissions as it proves it can be trusted. And that's the same onboarding model that we've used for decades with our own people.

Dustin Childs: Yeah, that's right, Johnny. I really appreciated that too and it was a good analogy. Another thing he said that really stuck with me is about how MCP servers really are a blind spot right now. And the way he framed it really stuck with me too. He said, until MCP looks like an API gateway with real role-based credentialed access, you need to treat every server you connect to like a brand new vendor. It's also worth remembering that TrendAI's own research found that real defects across roughly 19,000 scammed MCP repos.

Johnny Hand: Yeah, and it's a good call out, too, that security teams need a seat at the very first design session for agents, not just as a check before you get ready to launch them. And the OpenAI Hugging Face incident really showcased that once an agent has a goal, it will find its own way around any guardrail that you did not design. So I also know that we're both big fans of The Pit. I love that you did a callback there. And I also love the parallel with the show and how Zach's real life experiences in the hospital played out.

Dustin Childs: Yeah, without giving out any spoilers for season two, I mean, the hospital had some computer downtime and it really highlights his point that automating the work doesn't mean you stop needing people who can still do it manually. For example, Dr. Robby was able to handle it, but Dr. Whitaker was new and essentially was de-skilled to the point where he could no longer do his job. So, you know, the parallel works very well and it's very interesting to see that his experience in hospital really matched that of what the show did. And it's a reminder to plan for when the agent goes down, not for just when the agent works. We'd like to thank Zach Evans for joining us today. Check out the Show Notes for how to connect with him and learn more about Xsolis.

Johnny Hand: And that does it for another episode of AI Security Brief. We want to thank you for joining us. And as always, our goal is to host conversations to get you thinking differently about security. And if it does, consider subscribing so you don't miss what's next. >> AI Security Brief is mixed and produced by Elliott Peltzman, with original music by Amneajynx. Our executive producer is Jennifer Eiben, with content strategy by Ma'ayan Plaut and Melanie Galant. Additional production help by Liz Stokes. Video editing by Sarelle Joppy and Bridgitte Criqui Wild. >> Thanks so much for listening, and we'll see you next time on the AI Security Brief. [ Music ]