AI Infrastructure Debt: Strategies for Capacity, Resiliency, and Modernization

Summary

Isaac opened the episode on AI infrastructure debt with data showing that weak data foundations limit AI value and that many organizations lack automated, predictive infrastructure.

Liam explained that AI is primarily a data challenge and that enterprises often mistake it for a GPU problem, while the real bottlenecks are data movement, governance, and storage performance. He noted that AI-enabled environments can see data growth 2 to 4 times faster than traditional rates and that hardware scarcity is affecting capacity planning.

The group discussed the need for a hybrid cloud approach to maintain control and avoid vendor lock-in. Derek emphasized that AI magnifies existing infrastructure weaknesses and that resiliency must be designed upfront rather than added later.

Joanne highlighted the importance of human accountability and “do-no-harm” policies, noting that AI is not an on/off switch. Liz and Joseph shared that building a business case for modernization requires measuring current pain points and translating them into risk and missed opportunities for the board. Liam concluded that modernization should be positioned as a business acceleration strategy rather than a simple technology refresh.

Speakers

• Host – Isaac Sacolick
• Special Guest – Liam Yu, Senior Product Marketing Manager, Hitachi Vantara
• Digital Trailblazers – Derrick Butts, Martin Davis, Joanne Friedman, John Patrick Luethe, Liz Martinez, Heather May, Joseph Puglisi

Discussion

1. Capacity: What mistakes are enterprises making today regarding infrastructure for supporting explosive data growth and enabling AI opportunities?
2. Resiliency: What is AI infrastructure debt, and how does cyber resilience fit into AI?
3. Modernization: Where do organizations typically encounter the biggest bottlenecks, and how should IT leaders position the business case for modernization?

Research

Dashboard

Transcript

[00:00:00] Isaac Sacolick: Greetings everyone. Welcome to this week’s coffee with Digital Trailblazers, our 192nd episode here on October 16th.

So glad you’re joining us and sorry for again, a few seconds late.

Sometimes we just get a hiccup here and just have to roll with the punches. But we are here for our discussion today on AI infrastructure debt strategies for capacity, resiliency and modernization.

I’m thrilled to have Hitachi Vantara join us as a sponsor and our special guest from Hitachi Ventara, Liam Yu.

Let’s just get started and I’d like to share some data points that Hitachi shared with me, some fascinating areas in the area of AI infrastructure debt and we’re going to talk a lot about reliability.

$108 billion in wasted global AI investment each year tied to weak data foundations that keep 58% of U.S. and Canadian organizations from realizing value. Just a lot of money there. 84% say the complexity of their data infrastructure is growing rapidly or too quickly to manage, so we need to start catching up with the greater demands of data and AI.

And in fact, 84% of data mature organizations report measurable AI ROI, compared to the 48% of data laggards. Half say their systems are complex enough that executives would lose sleep if they understood the risks. That’s scary to me. And 43% report having predictive or automated infrastructure automations. And of course that means over 50% of you don’t. And so just a lot of opportunities to be looking at improving your infrastructure as part of working through AI and making that a strategic advantage in your organization. Now there are three areas that we’re going to cover today around capacity inference. Workloads account for roughly half of all compute in 25.

Deloitte says it’ll get up to 2/3 in 2023. It’s pretty remarkable that AI is going to be dominating our infrastructure so much in terms of resiliency. 80% of operating teams say their most recent outage was preventable and that better processes or management will have caught it. In terms of monetization, there are three tiers of data maturity that we all speak about.

41% qualify as optimized, resilient infrastructure, clean data actively powering their AI. 24% remain emerging who are risk averse, short on skills, maybe have some processes that won’t scale. And 35% sit in the gray middle making marginal progress without a strategy to break out. That will be our discussion today about how do we break out a strategy that basically said let’s keep the Lights on on our infrastructure in our data centers and for those of you in enterprises working through very large amounts of data and very much high demand for AI, we’ll be talking to you about how to resolve this again today. Before we begin, I’d like to thank our sponsor, Hitachi Vantara. Hitachi Vantara helps organizations modernize their data foundations so they can innovate with confidence. From enterprise storage and hybrid cloud to cyber resilience and AI ready infrastructure. Hitachi Vantara enables businesses to unlock the full value of of their data while maintaining the reliability, security and performance of today’s digital enterprises require. Thank you again. We appreciate the support of today’s discussion. All right folks, here we go. We’re going to be sharing lots of good examples around this today.

Liam, welcome to the floor. Welcome to the coffee with digital trailblazers. Welcome to our speaker advisors. Today we’re talking about AI infrastructure debt, strategies for capacity, resilience and modernization. Let’s first talk about capacity, Liam. You know, you get to see a lot of companies review their data infrastructure, see what’s going on.

What mistakes are enterprises making today regarding infrastructure for supporting explosive data growth and enabling AI opportunities. Liam, welcome to the floor.

[00:04:41] Liam Yu: Thanks Isaac. And thanks everyone for having me here.

It’s a very good pleasure. Yeah, capacity is definitely one of the first things we think through with customers a lot.

Infrastructure debt itself is really, when I speak about it, I really mean the gap between what businesses expect from AI and what the data, the storage, the resiliency and the operations of all of that actually support.

You know, so, you know, put simply, you know, AI is a data challenge before it’s a true model. And that’s kind of in the space that we’re in. I mean, there’s sort of that saying in, you know, a Chinese proverb may live in interesting times where we’re certainly in interesting times when it comes to capacity. You know, the first mistake though is treating AI as if it were a GPU problem, which is a natural thing to do given that we are modernizing applications. We’re thinking about the requirements of those applications from a compute perspective quite a bit. And I don’t mean to say that compute doesn’t matter, it most certainly does. But the real bottleneck is often access to data.

The data movement itself, the governance controls around that movement and usability and the storage performance and day to day complexity of managing storage performance.

These are the first critical success metrics that we start outlining with customers around AI. From a data foundation perspective, AI basically only delivers value when data moves quickly and safely across, you know, your, your environment, oftentimes that’s a hybrid environment.

You know, you, you must maintain access control. You must ensure your levels of sensitivity. Data are kept as they are in, in, in, you know, in, in movement and, and, and, and, and rest. And they must, we must keep approved locations in an audible trail so that a lot of the old, or I should say the traditional views of, of how we see data still exist. And our policies are certainly valid. We’re just dealing with a new template now.

[00:07:06] Isaac Sacolick: Liam, when we talk about capacity and AI, maybe break down a little bit of what’s driving the capacity growth.

Is it unstructured data? Is it running inference models on prem, what are you seeing in terms of why organizations are increasing capacity based on their AI investments?

[00:07:30] Liam Yu: Yeah, well, it absolutely is the inference in training and the consumption around that more so than any other. The modeling structures for AI.

There’s a lot of testing going on, of course, to develop and there’s a lot of let’s, you know, fail fast forward in the testing, if you will, to understand how AI is going to apply to us. But the other aspect of that is that we’re just simply under capitalized in a lot of organizations right now to measure this because it’s all very new. It sort of reminds me of the days of the Internet when we were trying to move from brick and mortar economics to online economics with all of the wonderful cost reductions and efficiency needs, but we didn’t really understand the systems that we were putting in place. And so that developed an awful lot of technical debt.

You know, the truth is that I can cut data prep, you know, down and if you have the right systems, you can do things. You know, we’ve seen, you know, Organizations move from 24 hours to 2 hours or less in terms of data prep. We’ve seen, you know, you know, million plus annual revenue increases to the models, but we haven’t actually, you know, a lot of organizations haven’t. You talk about the gray area, haven’t actually realized those things yet because we’re still trying to employ the best systems in place using older existing systems that we have. So modernizing that foundation is sort of like building a house. You need to lay a strong foundation so that your broader structure around application modernization and in fact the OSI stack itself is actually successful.

[00:09:16] Isaac Sacolick: You know, Liam, the picture you’re painting for me is that, you know, you know, the old model of what we were doing in our data centers was based on data science needs and application needs and Enterprise systems, file storage, you know, things that we have been doing for quite a bit of time now for about 10 years, with companies going back and forth of how much they’re storing on prem and how much they’re storing on clouds and what’s their resilience structure around it. Then all of a sudden, a couple Years ago, the AI boom hits, and now CIOs, CISOs, chief data officers, heads of operations are scrambling, right, just to keep up with what the board and what employees and managers want from just AI as a capability.

And now we’re backfilling into, well, how do we service this properly, how do we make sure that it’s performing well? How do we make sure it’s resilient, how do we move workloads more efficiently? You know, the demand isn’t just coming from one place. And the type of storage we need probably changes depending on the application. I think there’s just a lot going on there.

Less comments on this, on capacity, Liam, and then I’ll go to our board.

[00:10:35] Liam Yu: Yeah, no, I think you’re right. I think that, you know, what we’re really talking about is that, you know, AI is very ambitious, you know, and we see this as a.

Organizations see this as a competitive change that they must embrace. But, you know, we see, you know, things go sideways sometimes in early piloting. And it’s one of the reasons why we set up piloting early to do it, because we can see that the production of the infrastructure costs can rise faster oftentimes than businesses expect.

As they’re building these data pipelines, they don’t really have a sense of how much capacity they’re going to need.

As they’re scaling up, it’s important for them to move smooth as fast, I guess as fast as smooth, and not so quickly that they overrun their economic model.

[00:11:41] Isaac Sacolick: Thank you, Liam. Thank you for joining the group. Let’s just go to our speaker board, Joe, your hand raised up quickly to talk about AI debt and infrastructure. We have a capacity lag from where AI is driving organizations. What are you seeing? What are you hearing?

[00:11:59] Joe Puglisi: Well, I want to pick up on that last point that Liam emphasized. I agree with everything he said. Of course, it’s perfectly rational to, to look at your tech debt and, and evaluate where you’re at. But I do want to step back for a moment. You know me, I’m a fundamentalist, right? Always go back to the fundamentals. I really don’t think it’s a new problem. I think the capacity conversation we’ve been having for more than 10 years. Isaac. I think it’s been, you know, throughout my career.

Now it’s just centered around some new kinds of technology.

GPUs have changed some of the conversation from how many racks can I have to how dense can the racks be? And, and can I afford the electric bill with the consumption of these highly dense units?

I don’t think the discipline has changed. But to Liam’s last point, you know, follow the growth curve, right? Plan strategically.

I like to say we don’t buy a Ferrari to go to the grocery store, right Size it accordingly and plan, as Liam suggested, for the slow and gradual growth that’s necessary to truly support well, an investment in AI that’s going to give us some return.

[00:13:17] Isaac Sacolick: Follow the growth curve.

Joe, Liam, I want you both comment on this.

Gartner has said that the average enterprise average will have 150,000 agents running in a couple of years.

How do you follow that growth curve? Liam?

[00:13:38] Liam Yu: Yeah, this is what I talk about when I say they don’t really have a gauge for what the, you know, their, their rate of consumption is or their, you know, how they’re, how quickly they’re going to scale and grow.

But what we’re, what we’re finding is there are certain parts of businesses that are going to be consuming AI maybe a little bit faster than others.

What we recommend oftentimes is to map your model for consumption around the behavior of your business a bit more.

This is why we’re seeing over two thirds of mainframe customers that are now being asked to pull in from DevOps operations into an AI model, emphasizing that they co locate them, that they manage that structure in a very inspected way, that they move slowly before they just open the kimono up and start moving data across things again.

Most data is file and object. It’s unstructured. But what we’re now seeing is that there is a quick move to mount up, even block data to file systems and then vectorize the data into an AI model. So no data is sort of off limits, right. To an organization that needs or sees it as valuable to their competitive advantage. A lot of organizations need to move slowly so that they can measure that scale and then focus on the areas of their business or one at a time, if you will, or carefully rather than try to boil the ocean.

[00:15:21] Isaac Sacolick: I think we’re going to talk about this idea of flexible storage and realigning capacity when we talk about modernization. But let’s. Derek, you know, capacity has a lot of impact on our operations.

What are you seeing out there in Terms of what people are doing.

[00:15:38] Derrick Butts: Well, I see people and Liam had some great points there as far as the tech debt and the way the infrastructure is set up. You know, I see people trying to put AI now on existing infrastructure that already has issues. You know, sort of like I want to put AI make things faster, but you’re making it faster when you already have inefficiencies in the way your data is moving between your cloud, your edge locations and even the analytic platforms in which you’re trying to track it. The analogy I would use is like trying to run a marathon through rush hour traffic. It’s just not going to do well. And as far as, you know, the GPUs. The GPUs are supposed to help make things more efficient, but the problem is when you look at it, you’re putting GPUs on top of it as a fix, but you’re not fixing the pipes that already have issues. You’re not putting the AI governance in place, not in the security posture that’s needed, because the resilience piece of it needs to be there before you actually look at it. They’re ignoring the, as Joe mentioned, the operational type workloads of AI and what it really brings to the compute infrastructure. You know, the problem I see in the way this is set up, they’re not designing for cyber resilience up front, they’re throwing AI into a system that already has a bad data flow or workflows. And what happens is now you increase the bad, the blast radius of everything that’s going on now you get more access to potential compromise, more poison inputs. All these different things come into place as issues that people experiencing because they haven’t properly prepared for this, this push that we’re actually running into.

[00:16:59] Isaac Sacolick: Thank you, Derek. Let’s bring Joanne up. Joanne, we’re talking about capacity of our infrastructure, or data infrastructure in particular, to support what AI needs. What does AI need, Joanne?

[00:17:12] Joanne Friedman: Well, first of all, I would say what I’m hearing and what I’m seeing is the biggest mistake that organizations are making is confusing the capacity to store and process more data, ever more data, because it’s constantly increasing with the ability to create business value from it.

And so I would look, I look at it from the point of view that says, you know, more infrastructure doesn’t necessarily translate into better business outcomes.

So let’s be a little bit more careful when we’re doing compound city planning about not falling into three little rabbit holes. One is sizing infrastructure around technology rather than business workload.

The second is moving data unnecessarily because zero copy is a thing.

Many, many companies are using that. No etl, no elt. Let’s use the data where it lives, so to speak. And then the third one is ignoring the organization’s capacity to act on the intelligence, because every time you run an agent, you’re also running its audit trail. You’re also creating more and more data volumes. And a lot of companies forget about that part, that they’re bringing more to the table than they’re taking away from the table when they do the capacity planning. And so really, what I would say is, overall, you know, we spent billions making data available. It’s time to make it, make the data consequential now.

And so when you do your capacity, look at what really is going to be consequential to moving the needle forward in your business.

[00:18:53] Isaac Sacolick: Interesting. I want. Liam, I’m wondering if you have a thought on that. The first point, sizing based on tech and not business.

You know, how do you help organizations through that?

[00:19:05] Liam Yu: Yeah, I mean, she actually, Elizabeth, makes some very good points because again, we go back to where, you know, you can have an individual running multiple work workloads across different agents. And, you know, so, you know, you could have sort of agent inflation, if you will, in an organization. And that has, as Elizabeth said, a technical debt around all of it because you need to then, you know, not just, you know, manage that, that data flow properly, but also, you know, ensure that you’re getting the performance you need on the other side. So you have this sort of.

We kind of map things around sort of your innovation profile and your risk and cost profile, if you will, or your innovation productivity profile and your cost and risk profile and sort of how did that match together? We look at workloads very carefully. We look at sort of where the organization is going to be consuming things the most.

We start small with organizations or start with a smaller data workload set that’s highly prioritized.

The reason we do that is we’re looking to see where things break.

We’re looking to see sort of where the systems that are in place today and the policies will function well and where we’re seeing gaps because ultimately this is a technical debt issue.

And having the foundation right is, you know, is, is critical. It’s, it’s sort of like, sort of like you’ve seen the F1 races today. You know, how much time they spend on tires. Right. The engines are important, of course, but without good tires and without the right ones at the right time, you have some serious problems. Well, the same is fundamentally true with your data infrastructure foundation. And you want to start there so that you can build around the capacity thresholds, but also, you know, really identify where your gaps are.

And our real goal is predictability. Our real goal is, is, is to have a system in place that has some, you know, elasticity to it, but very much predictable in terms of how it’s going to behave as we add new capacity onto the systems.

[00:21:28] Isaac Sacolick: Yeah, both you and Joanne are talking performance as, you know, as just as much of a, or if not a greater importance when we think about capacity. Like how well is the storage going to perform when you have, you know, maybe 100 agents working on a couple petabytes of data in your context layer and now, you know, less than six months later, that grows tenfold.

Maybe you have the physical storage for it, but now you’re going to need to think about the overall performance around that. I want to bring John in. John, welcome to the floor.

[00:22:01] John Patrick Luethe: Hi Isaac. Yeah, thank you for having me here, Liam. As I go to upgrade the home lab and look at the Apple devices and the smart devices and I’m looking at the cost of RAM and I’m like, wow, that RAM has gotten really expensive. And I’m looking at the high end machine processing units and graphics cards and I’m looking at those things and I’m like, they’ve gotten really, really expensive too. And so what is happening for people that are ordering physical hardware for AI workloads? What’s happening, you know, what are you seeing with the cost? Like, has it gotten harder to order these things? Do you have to put your orders in a lot farther? And like, what’s, what’s the impact on capacity planning? So like, how far do people actually have to be thinking out? Like, like are they thinking out, like, is this something they can order and get next year or do they have to be thinking out farther? And then I’d love to hear about some wins that you’ve seen. So that’s, that’s what’s really top of mind for me, if you can share in those areas.

[00:23:00] Liam Yu: Oh yeah, yeah, no, both are great questions.

Yes. I mean, scarcity, if I could boil it down to one word, the demand is exceptionally high and it is causing an awful lot of backlog around the orders.

We’re going to see this for a while.

And so when we’re doing some of the capital planning with customers, we’re saying, look, you’ve got to, you probably are going to have to start Moving it out a little bit further so that you can sort of like an airline, buy the fuel ahead of time so that you will have the cost advantages today a bit.

The prices are going up, the availability is dropping fast.

And oftentimes AI or organizations around data growth often exceed the compute growth because they’re not, they’re not doing enough storage of the training data. They’re sort of not, they haven’t figured out how to deal with sort of the back end areas of capacity such as the data copy, the snapshot, the governance copies, the compliance, retention, all of those things. Right.

The bc and then so there’s sort of this need to sort of, you know, plan around a bigger data growth than you expect. On average, data growth in the past has been between 20 to 40% annually.

What we’re seeing with AI enabled environments, that’s at least twice as much, if not four times as fast in terms of growth.

So where we’re seeing successes are where we focus on a part of the organization to get the system down. Right.

And provide value upfront because you’ve got to be able to prove that value early in a capital plan so that you can get money later.

And you want to work with the business very closely. And so, you know, a lot of our architects are working very closely with how do we, you know, help in this particular area of the organization where we know we’re going to provide some immediate value and that we can also test the tires on the track bit and see that I mentioned, you know, some improvements around capacity. One of the, one of the, I can’t talk about the name of the company, but let’s just say it’s a very big retailer in the UK and they, they saw a reduction in data prep, significant reduction in data Prep under from 24 hours to less than 2 hours just by restructuring the data foundation the right way, not even talking about the AI modeling that goes on top of it. Right.

But that essentially fed them some quite a bit of success in the way that they could convert that into productivity around the model and it allowed them to test the model without additional cost impacts that they, that they identify forecasted. So it really does matter that you start with your data foundation first before you start tackling your application AI modeling.

[00:26:20] Isaac Sacolick: Liam, you’re getting a thumbs up from Derek. Let’s hear from Liz and then we’ll go to our break and then talk about resiliency next. Hi Liz, excellent.

[00:26:29] Liz Martinez: Thank you so much for having me.

I’m loving some of these points. I love that you have A focus on doing a small, realistic, you know, pilot or whatever you want to call it, to make sure that you have good business. Roi. I’m a big value realization person, as as many of our folks here know.

I want to go sort of from, from value realization, really. The other passion of mine is governance. And I, you know, as a value realization officer in my, my most recent role, I sort of view that role as a bridge between the CFO and the Chief Strategy Officer, that whatever is going on, there needs to be a bridge that says here’s where we want to go, but here’s really how we’re going to pay for it and how we’re going to be judicious and how we’re paying for it.

And my question to you is, where do you see the best way to govern this, like, infrastructure explosion?

[00:27:31] Liam Yu: Yes. And I, you know, I kind of brought that in too early on because it’s not just about, you know, making sure you’ve got that performance and success, but you really need to make sure you’re meeting your governance. This is, we are in unchartered waters and oftentimes a lot of our organizations are. This is, this is sort of their biggest concern is around not just their own, you know, their governance from a, from a, from a regulation perspective, but their own internal requirements for data. Right. Focus on areas of data quality and data ownership is critical. It’s obviously always been data lineage has been and classification is kind of a new set that needs to be revisited because, you know, we’re dealing with different types now.

Don’t forget, this is data now being consumed much more in machine fashion than it is in human consumption. And so we’re, we’re looking at, you know, sort of revisiting a lot of the data governance priorities. Right. What is, what is where. When we think about metadata management, where is it, you know, where is that data being, you know, moved and how is it being consumed and what, what operation operating models are doing so that have the right controls around them. Right. So the other priority, and we’ll talk about this, I think soon, is that, you know, security and cyber resiliency has to be brought to the forefront. These are zero trust architectures. They should be treated that way. Even though it’s coming in from our own internal dev, our application dev operations teams, we have to kind of say, all right, but we’re going to, we’re going to approach this as if it’s a zero trust architecture and design around that. We’re going to make sure that our, our model for security, access, governance controls, not just meet the standards for the individuals that are doing that, but the systems that we’re actually bringing in. And then we’re going to take a very close look, a very close look at our backup and data recovery systems and how we treat that.

We are very much concerned. We run these sort of small projects. Part of the early projects is the stress testing. But it’s also things like how vulnerable are we from ransomware in terms of the data that’s being moved out and used? And where can we ensure the recovery, not just the loss of that data, but ensure that protection is in place up front? Right, so a lot of this.

[00:30:01] Liz Martinez: I’m going to interrupt you. Well, I just did interrupting you, so let me go ahead and finish.

[00:30:06] Liam Yu: Oh, no, no. That’s the only other comment I would make is that we are also seeing a new trend emerge around policy internationally and that’s around privacy and data sovereignty.

There is a very, very big concern that, you know, we cannot hand over these systems to third party AI hosting. We have to ensure, you know, that not only we have the fidelity, we have the control around this. And so data sovereignty is becoming kind of a, a catch all phrase for effectively data control and making sure that you’re not losing control and you can’t attest to that, where that data is being being used and exposed.

[00:30:47] Liz Martinez: My follow on question is data quality, which is so important that I mean, it’s kind of also amorphous. And where do you see the governance of that data quality sitting?

[00:31:00] Isaac Sacolick: Let’s hold that question off, Liz.

[00:31:03] Liz Martinez: Okay, I interrupted.

[00:31:05] Isaac Sacolick: Yeah, no, it’s all good. I want to be able to welcome everybody who’s joined us midstream to our coffee with Digital Trailblazers, speaking today about our data debt and infrastructure strategies for capacity, resiliency and modernization. Today’s episode is sponsored by Hitachi Vantara. We’re discussing the opportunities and challenges CIOs face as AI adoption increases. It’s a conversation that aligns closely with Hitachi Vantara’s work. Helping organizations reduce complexity, strengthen resilience and build the modern day infrastructure needed to support AI initiatives at scale. Welcome back again, Liam, and thank you again, Hitachi Vantara for making this possible today and for being our sponsor. Folks, I’m going to be down in the Houston area on October 27th to lead this discussion in person at a coffee with Digital Trailblazers. We’re talking it will be a a roundtable with leaders on AI’s hidden data debt. And you could see the URL at the bottom of the screen starcio.com AI Houston Roundtable if you are interested in attending this event. And that event is also sponsored by Hitachi Vantara. Thank you for being such a strong supporter of the coffee with digital trailblazers.

Just to close out this discussion, Liam, on capacity, you know, whether it’s, you know, data prep, data capacity, lineage sovereignty, storing multiple snapshots and backups to protect against ransomware and that those are just like sort of the add on X factors that go on top of we’re storing more data, our AIs are creating more data. We’re building context layers on top of our existing data lakes and data storage. It’s just going to be a large growth and I am worried. We went from, you know, to John’s question, we went from a bottleneck in GPUs to a bottleneck in memory. We see all the data centers being built up. It’s just a matter of time between the next layers of shortages that we see. And for those of you who are going from hundreds to thousands of agents, I’d highly be recommending to look at your storage capacity and plans going into 2027. So Liam, with that, let’s switch off capacity. Let’s talk a little bit about cyber resilience and how that fits into AI and your AI infrastructure. Can you talk about cyber resilience a little bit?

[00:33:50] Liam Yu: Yeah. Well, here’s a simple way to think about it.

AI infrastructure debt builds. When AI ambition moves faster than the data storage and the protection system and the operational environments behind it, AI becomes business critical, you know, or as it becomes business critical. I should say that resilience is no longer just the IT concern, it’s the business requirement itself. Just following up on what Les said, you know, ownership of quality, as much as, you know, the, you know, how data resides is going to be tied directly to the business owners. That’s product finance, customer sales, hr, all of the places that. So there’s a partnership around that too. But resiliency is critical. Organizations must recover that trusted data and the AI services that they depend upon it without compromising the way that they’re copying or the violation of retention rules.

We see industries, what is it? The recent study, 8 out of 10 operational teams view their last outage is preventable, which is really a testament to bad planning in a way. Right. Again, why we do a kind of stress test around the track is in fact to break the car.

We want to reach the threshold, we want to see where it breaks, we want to see how it breaks and then we want to go back and improve upon those things. And so a lot of the, you know, piloting that’s being done is, is around challenging our assumptions around the system’s resiliency and making sure that we can, we can achieve better, you know, better goals than our expectations. We also need a higher threshold because our again coming back to, you know, today’s AI is not going to be tomorrow’s AI in terms of consumption. We want to know that we have a bit of headroom around resiliency.

Eight nines of availability is becoming more of a standard at the high end now more than anything else.

We see that as kind of a low bar threshold in some cases.

[00:36:16] Isaac Sacolick: Interesting. Derek is our CISO expert. We’re talking about his favorite topic of resiliency and data and now infrastructure. Derek, what do you want to share with us?

[00:36:27] Derrick Butts: Yeah, I look at when you look at the resilience and Liam was spot on. We talk about resilience factor and building these legacy networks because I do see it as AI infrastructure debt is really when you’re looking at, you’re trying to run these AI applications on an accumulation of shortfalls, legacy applications, you’re looking at temporary fixes. And all these things are never really designed for AI speeds, the scale at which we’re talking about nor the risk profile. And I think people are, the businesses are not asking the right questions. So when the data shows up and this resilience factor shows up, they’re having issues with the data breaking under load, they’re having issues with the storage we talked about because now they can’t do parallel process. The security controls are not in place because they assume humans will maintain them. Well now they’re outpacing what humans can actually perform. Also looking at the network pieces that they weren’t built for, distributed training, other things. And one of the things I talk about in several episodes is the monitoring. There’s no AI monitoring to see what’s actually happening in the AI workflows and that creates issues.

I see AI is just magnifying all these weaknesses that infrastructure currently has that now you think it’s okay. And I want to use the word that Liam talked about, assume. You know, we can’t assume right now. Trust and verify, the trusted network thing, these are things we have to go back and verify and check. Can it really handle this? Yes or no. And I think too many companies are assuming that they’re okay. They’re assuming that their resilience posture is okay and it’s in place until they Put the AI on it and find out it’s not. And, and by doing this, they’re really putting themselves at risk because now they’re depending on AI to kind of be the end of all to help them get through these things. And they’re finding they can’t withstand adversarial attacks, they can’t do with continuous validations, they can’t work with the survivor outages of corruption that AI brings to them. So I think when we look at this across the board, we really have to look at.

Resilience is not a nice to have, it’s a must. It’s a differentiator between how AI is going to be able to accelerate in your business and how can we actually avoid AI becoming the accidental attack surface.

[00:38:22] Isaac Sacolick: Thanks, Derek. I mean, Liam, I wonder if you can comment on this. You know, Derek mentioned monitoring and Joe and I were at a conference yesterday where we heard a really smart lady asked the question, are we building AI into our doctor plans now that we’re becoming more dependent on all these agents and models? Can you talk more about not just the physical, hey, I’m putting disk in place, but. And we have the ability to manage capacity, can we talk about monitoring and what people should be looking for there?

[00:38:58] Liam Yu: Yeah, we’re seeing an awful lot of. This is actually a really important thing that Derek’s pointed out because we’re trying to figure out, we want to observe the results in the testing that we do and then sort of make advanced planning activities around those. I mean, predictability is very important. So at three in the morning, you know, yes, it’s important that, you know, you get an alert that something’s happened. But more importantly is that the, the advanced planning that you’ve put in place was implemented and that you had, you know, protections, your policies and your were executed with policy controls that you can go back to and have records. So monitoring the data is critical. But I would say really we’re seeing more emphasis around how do I ensure that the advanced response around the system behaves in a predictable way to ensure my data has been protected and that my system is up and running and that I have sort of attestation to that business continuity.

And so there’s sort of a automation aspect and a predictable automation aspect that’s been well thought out around that tactic that the monitoring reveals. Right. And early monitoring of the training systems or the testing is important to then map those things out.

It’s a little like how the military sees that no plan, you know, you know, survives the enemy. It’s a similar approach. Right. You need to have kind of those, those well defined and predictable contingencies in place so you can go back. The other thing I would add is that when it comes to the data of AI, it’s not just protecting the production data and ensuring the fidelity there. It’s also the training data sets, the vector databases, the prompts, the data pipeline itself. Right. So there are these sort of broader sets of data of the data system that you need to not just monitor, but have kind of predictable outcomes to ensure that continuity.

[00:41:17] Isaac Sacolick: Thank you, Liam. Let’s bring Joanne back. Joanne, we’re talking about cyber resilience of our AI and data infrastructure. What do you have to share for us?

[00:41:27] Joanne Friedman: I have a couple of points that I want to make. First of all, I think for boards and executives and people who are really thinking strategically about their use of AI and what it’s going to bring for them going forward, the critical issue from a cyber resilience point of view and also general resilience point of view, is ensuring that they change the mindset around.

I can kill it with a switch. AI is not an on off switch, it’s more like a gate. It’s got a floor and it’s got a ceiling.

And understand that there are constraints and boundaries that have to be built in and thought about before they’re built in, long before you deploy. It’s not just the data and the provenance and lineage of that data to give traceability and explainability and observability.

It’s beyond that. It goes to what is the revocable and irrevocable authority and human accountability that you’re going to build in from day one, because the predictable outcome, there is no such thing. There’s always exceptions. And no matter how good the zero trust architecture is, somebody’s always going to find a way to try and breach. Coming from manufacturing, which is the number one industry to get hit with cyber attack, and has been for the past three, almost four years now, we’re talking about human life as a risk.

So for me, the biggest risk is what happens when AI makes a bad decision.

It’s not just about the data infrastructure preventing it. It’s about using that infrastructure to embed the policies that say, do not ever do this.

And that should be the first line on any prompt, whether it’s a generative AI or an agentic. Do not ever, because they will always find a way around you. We’ve seen this in, you know, the news of late in several occasions.

It’s how really determining what is the most severe risk that you have to deal with?

In, in some cases that could be human life. In other cases it could be, you know, decimating an organization by having all of its data removed or considerable amount of it put out into public.

That authority to act means you also have to have a way to intervene.

So it’s what happens at the exception of Prediction Land.

[00:44:07] Liam Yu: Yeah, you’re talking, you know, it’s interesting, you’re talking about the economics of that are emerging around AI, around accountability.

You know, one of the things that I’ve been hearing a lot is that, you know, AI is going to take on a lot of these sort of perfunctory fact actions. But, you know, at the end you need a human being, even if AI could run the whole thing because you need accountability. You need to have kind of, you know, override control and you need to, you know, approach the problem of predictability that you’ll, is that you’ll never achieve it. Right? You, you want to pursue it. But again, no plan survives the enemy.

So you’ve got to think about it in the context that AI is more like power steering than it is about auto driving. It’s really about trying to make sure that human beings have degrees of freedom to control things. I’ll give you another example. These always on or do no harm models that you speak of are very important. In fact, policy for a lot of organizations in healthcare, even in finance, restoring fraud detection is unacceptable at 15 minutes, let alone four hours.

They want it down to under. A second perfect example, and this ties into the who owns data quality scenario, is that we have seen a scenario with financial institutions, for example, that when someone passes away and their Social Security is cut off and they, they, you know, they, they. But, but it keeps, keeps, it keeps coming into the bank. The bank actually does not want to be accountable for the Social Security deposits that are actually valid because then there’s a lot of clawback and costs and all these other things, let alone the, you know, potential legal liabilities around this. So they wanted, they want to design a trigger that ensured that that policy is met immediately and that accounts basically cannot be accessed once that recognition of that Social Security deposit is being requested back.

And so it’s almost instantaneous.

Having said that, the organizations want override, they want to make sure that if they need to pass this to a legal observation or they need opinion, they need to have some kind of policy around that, that they can do it in a very efficient way and they can account for that activity. So this is Just like a real life scenario that’s really about the business, but it’s a do no harm scenario and that needs to be done within seconds with manual oversight and observation around that, that, that tactic.

[00:47:02] Joanne Friedman: And, and I, I would add to that that this is why cyber resilience has to evolve beyond data and, and even to the point of, to the hardware of a remote control kill switch. You know, if worse comes to worse, unplug it because it is a mechanical device after all. And a lot of companies are beginning to look at how do we extend cyber resilience into our hardware planes, our data planes, our policies, our embeddings, not just the models that are being used or the harnesses that are around them, but the physicality of it as well, because that can be in some cases the only way to intervene.

[00:47:50] Isaac Sacolick: Thank you Joanne and Liam, let’s move over to John and then Liz, just hold off. I want to get into our third question before I call on you because I think you have something to say about business case. Go ahead, John.

[00:48:03] John Patrick Luethe: Yeah, and Liam would love to get your thoughts here. Like I spent my life, you know, the last 20 years trying to get people out of data centers. I was even at a company that built data center equipment for people, but I was on the team that was getting people out of data centers and I would love, and that data up to the cloud and I would love to hear your thoughts on like what’s really bringing people back back to, on prem and like, you know what, like you know, is, is there anything in the cloud that’s particularly causing people to, to want to put their, their, their assets back on prem versus having their AI assets up up in the cloud. And I would love to hear your thoughts on the decisions, the security aspects of that and, and the outage aspect too because it’s, it seems like when there’s a global outage, it, it just, it impacts everyone.

[00:48:50] Liam Yu: Yeah, this is sort of why we’re seeing a return, return to data center is, is really about a smarter hybrid cloud approach than in the past.

There are a number of concerns.

One concern is that these models and economic model, not the technical model but the economics around putting everything in the cloud have kind of folks are kind of consciously aware that that, that, that can result in a lot of vendor lock and loss of control around and certainly even though your contract might say oh certainly you can stop buying from us, but you’re going to have to deal with the data ingress and egress costs associated with moving into a new model that’s an economically prohibitive alternative that they’re very uncertain about.

So the first thing is probably an economic concern.

We want to use cloud. We need to use cloud properly and we need to kind of design around a hybrid. This is an important way to think about the new world order. But it’s not everything in the cloud and it’s most certainly not.

It’s going to be dealt with based off of how much exposure and cost does my organization have to will tolerate. Right. The other, the other thing is that you have many more tools now at your disposal to create your own kind of hybrid environment and control. That data and control is really the, you know, the subtext of sovereignty. Right. It’s I need to make sure I’ve got control so that when the policies change down the road and all of a sudden I can’t move data out of, you know, from, from one point to another because of some governing policy around that data that I can pivot.

So a big, the other part of this is I want the flexibility to have my data in control in different ways and I want to ensure that I can employ new tools that I haven’t had in the past that will help me accelerate and predict better how that data is going to be used and ensure that that resiliency and I’m accountable. That’s probably the last aspect of this, is that we’re going to enter into accountability economics, right? Which is I can do all of this stuff, but if we don’t put a person at the, at the helm, then we have no real way of measuring the, the, the, the accountability and the risk around this. And we don’t have, you know, a group of authority that we can go to and help remediate or correct the system. So, you know, we’re trying to build systems that have predictability but also have control.

And having kind of the renewed data center approach has sort of been the way that folks are thinking about it now is I want my cake and eat it too around hybrid. But I want to ensure that I’ve got control to pivot and I’m not locked into some downstream change in the terms of my agreement with my cloud provider.

[00:52:15] Joe Puglisi: Thank you.

[00:52:16] Isaac Sacolick: Thank you, John. And thank you, Liam. Let’s do a very rapid fire on our last question. We’re going to go to Liz and then Joe and then back to Liam to close out.

I want to talk about building a business case for modernization of your storage infrastructure from the perspective of that very frustrated head of operations, head of infrastructure, head of security, who sees all this effort, work and dollars being put into doing AI and now has to raise their voice up and say, look, we’re sitting on some broken pipes, to use Derek’s analogy, and we need to not only improve resiliency, but we need a lot more capacity.

Liz, about a minute. You got what, how are you making that business case?

[00:53:05] Liz Martinez: Oh my God, you’re talking my language.

First of all, you’ve got to assess how bad things are, hopefully rather quickly, right? Understanding what you’re actually sitting on, where you’re maybe where your holes are in terms of resiliency. So there’s, there are, it’s quite a complex problem because you have to gather information from many places like where your, where’s your spend going, what do you actually have inside? What do you have in terms of legacy, what do you have in terms of your data quality to some of these other points and really take a wholesale look at all the different aspects where the modernization might be beneficial. You got to sort of start looking at all these different pieces, including roles and responsibility and governance, etc. So it’s not a small problem to create that, that business case, but once you start gathering those pieces from the different areas, the fact that the business can’t get the information that they want, the fact that, you know, we’re spending too much money in storage, the fact that, you know, we’re having infrastructure, our pipes are old, etc. We don’t have good resiliency. Once you pull all those pieces together, you can then say, okay, we’re going to make the case that if we actually do some modernization like this, this, this, this NEO cloud hybrid structure, or we do some data cleansing, or we implement some better governance and some better resiliency.

This is what we’re going to have in terms of speed to market, this is what we’re going to have in terms of being able to make decisions faster.

[00:54:42] Liam Yu: This is what we’re going to have

[00:54:44] Liz Martinez: in terms of being able to access new markets or, you know, actually improving our margin. If we just clean up Iraq, we got to clean up the garage to even find out what we do or don’t need and then, and then buy some new tools.

[00:54:59] Isaac Sacolick: Joe, you got a minute to convince the board that they need to clean up the garage and fix the pipes? What are you saying to them?

[00:55:06] Joe Puglisi: I’m going to speak in, in the language they understand. The most important thing here is to understand the industry you’re in and where the board’s head is at. So I’ll give you an example. Recently, an Engineering firm discovered that you can’t add 10 more stories on top of a building unless the supporting infrastructure of the existing building will support it.

I worked for a construction company several, in fact, 16 years in that industry. And so you have to put things into the parlance of the board, you know, things they can relate to.

This is exactly the same situation. If you’re going to put 10 AI stories on top of your existing foundation and your existing foundation cannot support that weight, the building is going to crumble. They’ll get it.

[00:55:57] Isaac Sacolick: I’m hearing from Liz, measure the current pain points. I’m hearing from Joe, translate those pain points to both risk and missed opportunities. Liam, what did we leave out in terms of our business case for modernization?

And are there any, any bottlenecks organizationally that you want to help our heads of operations through?

[00:56:17] Liam Yu: Yeah, I think Liz nailed it.

Modernization is a business acceleration strategy. That’s how to approach it. Now, we all know that there’s a lot of opportunity to reduce costs and improve our controls around things, but fundamentally, we want to sell it as a business acceleration strategy. It should not be positioned as a technology refresh.

Avoid that in. Put that in red. It is, you know, never approach it that way.

The complexities that we’re trying to deal with are a lot of things that, you know, you guys just talked about. The silos, the manual work, the operational oversight, the advanced planning to achieve a level of predictability that’s much higher. And the most importantly, you need to have headroom. Because unlike other events in our past 26 years of change, this one has the ability to accelerate very, very quickly. And it’s about those workloads and those consumptions and the fact that one person can have 18 agents running at a given time, and so he can get away from you very quickly. And so what we want is we want to build for that advanced capacity planning and resiliency planning up front to ensure business acceleration. And what we do oftentimes is we’ll walk through, we’ll start with the business, and we’ll tie that to workload. So, for example, we’ll go to a manufacturing, we’ll show a use case around manufacturing, and we’ll start with sort of AI provisioning reductions of three weeks to three days, you know, and an estimated, you know, actually a resulted actual 200k annual savings.

How do we do that?

And then we work backwards from the workloads to show how it’s tied to this data foundational structure that allows the organization to try different AI modelings as they move along and find the best one. And ultimately that’s sort of the world we’re in right now is that we need the organization to get on the field, get lack of a better word, get bruised a little bit so they can get their plays down around AI. But they got to have the gear and they’ve got to have the equipment and they’ve got to have the foundation in order to do that well. And that’s really about accelerating the business. Smooth is fast. Fast is smooth.

[00:58:52] Isaac Sacolick: Thank you Liam and to all my advisors for joining again today. Thank you for listening.

Thanks again to Hitachi Vantara for supporting supporting Today’s Episode. As organizations continue their AI and digital transformation journeys, success increasingly depends on having a modern, resilient and intelligent data foundation. To learn more about Hitachi Vantara is keeping enterprises prepare for the future of AI and data innovation, visit the resources linked with today’s Episodes. You could find those in the Common Stream. If you’d like to learn more about the challenges discussed today, Hitachi Vantara has published insights on navigating AI infrastructure debt, modernizing data platforms, and building resilient AI data architectures. You can access the resource link using the links in these podcasts. If you have any trouble finding those links, just let me know. I will send them to you by message. If you are in the Houston area and want to join us on the 27th, the talk about AI hidden data depth, we’ll be doing that. You can join. Sign up for at starcio.com/ai/Houston-Roundtable thank you again Hitachi Vantara. Thanks Liam for joining us and everybody. Have a great weekend.