Change Management: Why AI Adoption Stalls and How to Measure Progress

Summary

Isaac shared research showing that while many employees use AI, few report it has meaningfully changed how work gets done, and many managers do not support its use.

• Sandy argued that adoption stalls because companies are not redesigning workflows or org charts to fit AI, but are instead bolting tools onto old processes.
• Liz emphasized that AI initiatives must align with clear business strategy and that the “why” must be communicated to impacted customers and employees.
• Martin noted that the rapid pace of AI evolution makes it difficult to maintain a stable strategy and that leaders should focus on avoiding major mistakes.
• Derrick highlighted the need to ask upfront questions about workflow changes, accountability, and guardrails rather than addressing them after deployment.
• Joanne suggested using a “Five Whys” approach to identify operational bottlenecks and execution gaps before applying AI.
• John warned that speeding up one part of a process without addressing the whole system creates new constraints and that AI-generated code can lower quality if not managed.
• Joe stressed that the board and senior management must personally experiment with AI to understand its capabilities and limitations.

The group also discussed the importance of moving beyond consumption metrics to measure actual business value and the need for continuous feedback loops and collaboration.

Speakers

• Host – Isaac Sacolick
• Special Guest – Sandy McCarron, Chief of Staff, VirtusHQ, CEO of McCarron Media
• Digital Trailblazers – Derrick Butts, Martin Davis, Joanne Friedman, John Patrick Luethe, Liz Martinez, Heather May, Joseph Puglisi

Discussion

1. Most employees now use AI, yet few say it has changed how work gets done. What’s really stalling AI adoption aimed at meaningfully reshaping operations — even after leaders have communicated objectives, values, and what’s in it for employees?
2. If consumption/usage metrics aren’t sufficient, what should you measure in the early weeks of a pilot or 30 days after an AI deployment that tells you business outcomes will follow at scale?
3. If you were starting an AI change management program, what would you put in place first before rolling out AI tools, pilots, or other initiatives?

Research

Dashboard

Transcript

[00:00:00] Isaac Sacolick: Greetings everyone.

Welcome to October 9th, our 191st episode of the Coffee with Digital Trailblazers. We are now officially into our fourth year of doing this together and it’s been a fun ride. We’ve had some recurring topics.

Data governance we’ve covered a few times and now change management we’re going to be covering today and probably ongoing into the future.

Change management has always been one of these areas that organizations have lagged behind in their transformation programs under invested in it, in their transformation program, sort of brought it in. Hey, we’re ready to deploy now. Let’s get people involved.

And I think most transformation leaders and executives have seen past that.

I think there’s a much greater emphasis on that today. In fact, I was just coming, I’m just coming off going to WebEx1 in in Austin with the group from Cisco and you’ll see in my article coming out on Monday where I asked them about why aren’t we seeing enough transformation activities around AI? Why aren’t we transforming enough?

And this CEO from Glean talked about it, the head of the president of Cisco talked about it. And their answer was that you’re not driving enough culture change. And culture change actually has to precede transformation when it comes to AI.

But I’m going to share my research slide now. And this was an insight I got from doing this article. It was published this week. It’s called 5 Metrics to Prove youe AI Strategies Business Value. It’s out on CIO.com if you go to CIO.com, you’ll probably see it somewhere on the homepage.

And you know what was apparent talking to experts about getting to AI business value was that adoption was being mishandled or mistalked about as if adoption was the business value.

And encouraging leaders to look past adoption and saying okay, you will eventually get people doing things with AI, hopefully aligned with strategy, but hopefully, but more importantly targeted toward things that will actually deliver business value. And so the article lists five different areas, five different targeted outcomes you can target from growth, customer retention, employing capabilities beyond productivity impact to customer and employee experience and then other operational impacts. I shared data points and some some research grounded in that. And then like all other things when we talk about outcomes, we better be able to measure that.

And I shared a number of different ways to measure these different outcomes that you can target. So adoption is a step in the direction of getting to these biz to this business value. And that’s what we’re going to home in on today, right? We’re going to try to look past some of the easy things that we know about that need to go into change management programs and talk about what you might be missing. Giving you some frameworks and some advice around how to arrange your change management programs so that they are stepping stone into delivering some stats here around why adoption stalls 64% are using AI, only 22% use it daily and only 14% strongly agree it changed how work gets done. 36% say they strongly agree their managers support their team’s use of AI. My problem with that is that means 64% don’t. I mean that’s problematic. How are you supposed to grow adoption if your managers are not leading by example? Frontline workers using AI daily or several times did jump from 50 to 74%. And I do like this quote from Satya Nadella from Microsoft. We have to get to a point where using AI to do something useful that changes the outcome of people and communities and countries, trees. And I translate that to aim for something big, aim for something impactful, change your culture to drive a greater mission and then what I left here I didn’t want to give everybody answers on how to measure progress. I’m hoping Sandy, Liz, Martin, Derek and all the other speakers here will give us their favorite metrics they’re using to measure the progress of adoption. But here are some things I’m recommending not to do. I mean training completed, that’s learning, but not necessarily due doing your licenses, the number of AI agents deployed to production.

These are all consumption metrics. They’re not necessarily telling you whether people are doing anything that’s meaningful.

If you talk about launching AI in workflows and talking about old metrics like on time, going live at a specific point, these are meaningless.

We want to make sure that faster time to value and addressing risks is what we’re really looking for here.

We talked about agents in production and fewer help Desk issue tickets does not speak to quality of results and customer satisfaction. So you know, wrong metric, wrong outcome. Wrong no wrong outcomes that you’ll get to so this slide is available at drive.starcio.com Coffee you can also get there from starcio.com Coffee Episode 191.

All the links in here are hot links so that you can get to your sources. And do visit the page to see my article on 5 metrics to prove your AI strategies business value so today our conversation is really going to start with our special guest, Sandy McCarron. Sandy’s been here before.

Sandy is featured in Digital Trailblazer I won’t say what chapter and what her pseudo name is.

She is the backbone of some of the Agile programs that I talk about in driving digital.

Sandy, welcome back. I’m going to give you a chance at, midway through the program to talk about what you’re doing nowadays. But you know, most employees now use AI, yet few say it has changed how work gets done.

I’m really interested what’s really stalling AI adoption aimed at meaningful reshaping of operations.

And I want to get past some of the easy, low hang fruit, right? If you’re not communicating, if you’re not specifying what values and outcomes you’re trying to do, if you’re not defining what’s in it for employees, start there. All right, so none of my speakers can go into those areas. I don’t want to talk about communications. I don’t want to talk about, hey, you have to have an answer for business value and communicate that to employees and what’s in it for employees. Let’s get into some of the things that we have to do specifically around AI that’s going to lead adoption into outcomes. Sandy, welcome to the floor today.

[00:07:55] Joanne Friedman: Hi.

[00:07:55] Sandy Mccarron: Hello everyone. I’m going to just start off with, I don’t think that we’ve actually redesigned the work. I think people are adopting a tool. I think, you know, everyone uses chat, GPT. We know how to make new decks in five minutes and no one is actually redesigning the old org chart, figuring out where there’s, you know, a process that could get redesigned that actually removes work as opposed to, you know, just kind of getting it bolted onto the current work.

There’s not a ton of traction on, on how they’re measuring the work that has been changed. Where’s the time go if I save a whole day using AI, what am I supposed to do with that?

I don’t see where, you know, leadership is not stepping in to say, with your save day, you’re, you’re going to go do R and D and figure out what our next opportunity is now that we have this faster 10x employee.

[00:08:59] Isaac Sacolick: You know, Sandy, that, that those are really good points and I’m trying to contrast that with the data point I just shared that 64% are not even encouraging their staff to use AI.

How the heck are they supposed to redesign work or read it, recommunicate what people’s jobs are going to be if they’re not involved? Is this a middle management problem?

[00:09:22] Sandy Mccarron: I absolutely think so. But I think it’s also a CEO program having gone through this type of transform multiple times in my career. I think a big one that I worked on was working at Random House when we, the advent of online booksellers was coming to, you know, coming to life. We had to look at all of our internal systems which would now be externally facing because it was a bibliographic data, ISBNs, you know, people were not having their, their author’s names or book titles in the same format or any way that was data based. And of course we didn’t have AI in the late 90s, so it had to be done by hand.

So a lot of the cleanup work that is a precursor for AI is, is a lot faster now. So getting your data sets cleaned and you know, we didn’t, we had to actually do that with human beings at one point.

The one other advantage we had there was that we had, we were sat on the executive floor. We, the entire company knew what we were up to.

There was not a ton of resistance when we made a decision that was bold and impacted workflows and how things were were happening internally.

And you do bump into big organizations internally like the sales team.

The sales team was not interested in, in a online bookseller, even if it was Amazon. They wanted to kind of maintain their, their structure and keep their accounts happy. And by us putting all of these different new technologies in place, it was going to change. And not everybody is comfortable with change.

So you know, you got to have somebody at the top saying that this is the mandate.

It’s going to be, you know, big and it’s going to be impactful. But then you’re going to, we’re going to figure it out what the new work is and what the new roles are. We didn’t have a concept of a product manager 20 years ago.

You know, it was, it’s a fairly new, it was a fairly new concept to come up with like different roles.

[00:11:37] Isaac Sacolick: Thank you, Sandy.

You know, Liz, it’s very interesting. When I spoke to the folks at Cisco, they sounded like they went through a bottom up plan, right? Their initial plan was let’s get everybody free use of AI. And they did that last year and then this year they gave everybody an AI assistant. But it was very much a top down strategy, was let’s break the fears and give everybody some freedom. And then they slowly rolled in.

You know, here’s where we want to approach this.

They later had a conversation of you need to prove to your manager what value you’re getting from that AI. And now they’re following through on one of your suggestions. Liz, they’re actually giving everybody a budget and that budget is going to increase with demonstrable increased value. So I mean, Liz, it was a very bottom up plan, but a top down strategy over at Cisco.

You know, how do we get adoption that’s really going to point in the right direction?

[00:12:41] Liz Martinez: Liz, I love, I love that approach because it really acknowledges the fear that’s already out in the workplace to try to just get people put hands on keyboard and actually trying it and actually giving them some support and becoming more comfortable with the tools.

I love that and, but I have to say that the way I think about AI initiatives is very much like business cases and the business value that any AI is going to be, you know, delivering. Because I think of AI as the tool, right?

[00:13:20] Joanne Friedman: It is a tool.

[00:13:22] Liz Martinez: So you know, if you think about it that way, like any other a, any other business change initiative, you have to make sure that the, you know, you don’t want to use the word communication, but that the why, the why is very clear and that you actually are, are measuring the feedback from the impacted customers, whether that’s internally or externally, on whether or not you’re actually getting the why addressed that you wanted to do. And that could take, that could take many, many forms. I’ve seen, you know, something as simple as, you know, we’re doing this initiative to make sure that it’s easier for the front line to do their work.

So let’s ask them if it’s actually making a difference.

You know, let’s ask them if it’s actually making any kind of difference. I would argue, by the way, that you said something about on time delivery is not a measure of success. It is not completely meaningless because if you actually get to on time delivery, it also means that maybe you’re not throwing good money after bad.

[00:14:29] Isaac Sacolick: I think it’s fearless. I mean, I think you raised some questions around that. I think we might have to have a separate debate about how we position AI, whether it’s a tool or an assistant or a partner or something that’s going to take your job away. Because you’re right, the fear is out there.

It’s particularly out there in companies that are, you know, very digital driven and IP driven. Hey Martin, I’d love to get, I

[00:14:59] Liz Martinez: just have to say on that point, these initiatives need to align to strategy. If the strategy is not clear, then the AI initiative is not clear.

[00:15:09] Isaac Sacolick: So I’ll tell you it’s not happening and I’ll tell you why.

It’s because mostly what’s happening is that the tools have been put out there long before the strategy was communicated. And even when it does is changing every three months. And even when that happens, the tools around getting a roll up of what all people are doing in a department around using AI and all the agents they’re experimenting with, there’s just, I have not seen a tool that does a really good roll up of this yet. And it should be possible, right? Roll ups are summaries and summaries are AI. I mean, so we should be able to get a pulse of what people are doing with AI. But I haven’t seen the tools do this. Martin, I’m reading an interesting book.

I’m going to preface this because I know you have a lot to say on change management. The book is AI Native.

It’s from Kai Fu Lee.

He’s gotten a few really good books out there around AI and we’re going to have a separate conversation around this. But he’s pretty critical about CIOs leading anything around AI and sort of boxing us into the, you know, as operators and managers of risk.

But he does have a point. His real point is the same that Liz just said, is that the CEO needs to own this and step into a much more operating role to drive change than probably any other program that we’ve seen over the last 20 years. So with that, how do we define, how do we avoid stalled AI adoption and point growth and change management in a way that delivers or at least is pointing in a way that’s delivering business outcomes? Hi, Martin. Hi.

[00:16:56] Martin Davis: Well, I’m going to take this in a couple of different places to start with. Anyway, I said at the moment, trying to control AI is a bit like grabbing a tiger by the tail, as in you’re just going to be along for the ride.

And I think that’s part of the problem, is change management takes a lot of time and effort. Change management is not something you can just do. You do not just change culture. And the problem is that AI, yes, it’s a tool, but it’s changing faster than anybody can actually keep up with.

So there’s no wonder no one really has a workable strategy or a strategy that anyone’s actually going to believe.

And it’s no wonder that the culture doesn’t know really what to do about this because it is changing so rapidly. And yes, we’re all using different tools or, you know, quite a few people are using different tools, although some of those numbers there were interesting.

But it’s a case of this is something it’s Almost impossible to get ahead of.

And therefore your culture doesn’t know quite what to do. So your culture is going to be all over the place.

So your AI strategy in inverted commas is going to be all over the place. And therefore how do you actually know what to do? I think at the very best you can actually kind of try and shape things and push things in certain direction and try and avoid certain things. So you can try and avoid data loss through AI gateways and data loss prevention measures and things like this is trying to actually avoid the biggest mistakes I think at the moment is the best you can do. So there you go. I’ll throw those out there and take a closer look at the LinkedIn comments because I’d be interested to hear what people think about those statements.

[00:18:50] Isaac Sacolick: Yeah, it’s a great comment here from Tony. A big mistake that many organizations make is that they take a one size fits all approach with employee adoptions. And you know, that’s not a new problem, it’s not a new issue. I think the real issue is that the platforms and the technologies are changing so much. I think it’s really hard to individualize the experience and particular since it’s not just about a business strategy, it’s really about that role change, that responsibility change that we’re all going through and how people feel about that. What do you think, Derrick?

[00:19:35] Derrick Butts: Yeah, I mean it was spot on. I mean the, the comments that Sandy mentioned about, you know, looking at the adoption of the tools and the redesign of the work, I think that’s the missing part. We talked about the business culture and the problem is you downforce artificial intelligence applications into old workflows. So the business culture has to change the operating model in order to do this adoption. And what I’m seeing is they’re not asking the right questions. The bigger problem is, you know, you’ve got all these new things you want to do with artificial intelligence but using yesterday’s processes. So in looking at this, I think, you know, the questions need to be asked is, you know, upfront because you mentioned they’ve already deployed the tools and now they’re trying to wrap AI governance around it and figure out what the risk they’ve exposed. But they’re not really asking the questions up front. Where specifically should AI change my workflows? What decisions can AI support while having some sort of human in the loop? What’s the accountability I get goes to if somebody something goes wrong? These things are not being asked up front, they’re being asked after the fact. And that’s where the problems come into this. Because now you’re having systems that are AI is not available, it’s compromising intellectual property, it’s producing unreliable results. All these things have not been thought out this after the fact. So when you’re looking at this, AI is one of those things they see as another tool, but it’s also another burden because it hasn’t been fully thought out as far as what they’re supposed to do. And because of that, transformation stalls. If you’re going to adopt this and do it in an efficient and safe manner, we have to establish trust. And I think that’s something goes with Liz mentioned earlier. The strategy has to be in place in order for the employees to adopt this, understand what they need to do, where does it go and how to validate the outputs when we actually put it in place.

[00:21:13] Isaac Sacolick: I think that’s really important about.

I’ll let someone else comment on trust because I think that’s part of the issue here is sort of like every time, you know, we start using technology as a catalyst for doing things differently, there’s going to be a group of people saying, I’ve seen this before. And in terms of relaying strategy, I mean, Wayne Anderson is talking about this a little bit in the comments stream. You know, I think the biggest problem organizations are not communicating is a path to customer impact. And I’m talking about buying customer impact, I’m talking about growth, I’m talking about customer experience, you know, and then, you know, being upfront with everybody, that every workflow is going to change, everything’s going to have AI in and it doesn’t mean it’s going to be fully autonomous.

I think we’re going to have our discussion in our debate in a couple of weeks about which of our agents and our assistants are going to be autonomous and which are going to be more human augmented.

But if you go back to people and say, look, these are our most important workflows, they have to change for AI. This is how we sell things to customers. This is some of the things we’re exploring around AI and, you know, using those as guiding posts. I just don’t think companies have done a good job communicating that. Let’s go to John. I’m going to just bypass you for a second. Joanne has a hard stop at 11:30.

Joanne, you have the mic.

[00:22:41] Joanne Friedman: Thank you.

This is an operational issue and you know, I hear many companies say over and over again, oh, we want to be AI first. Oh, we want to do this with AI. But in actuality, what they’re not doing is going to their employees and saying a what is the part of the job that causes you the most grief?

Not from a workflow perspective, but from a human perspective to see where the hook to enjoying collaboration with the human workforce is. And number two is in that operationalizing and changing their business models, which is really what this is all about. It’s not about the tool set or people’s use of the tools, even from a budgetary perspective. It’s about what can they improve that ultimately becomes like dominoes and segues all the way through to the customer that they are selling to. If you look at the customer feedback and you look at the human workers feedback about their jobs, about their roles and about how things run, that’s when you use a five wise methodology.

Understand really where the bottlenecks are or how to make people’s lives easier. That’s CEO down. Absolutely. As an initiative, it’s, it’s now a corporate initiative to do this. And then you start to flesh out where AI will be most helpful or not work at all. Because if you already have a lot of automation in place, you may be complicating matters by using AI as well as the automation you already have. It’s not a panacea and I’m a very big proponent of the, you know, go back to the basics before you start instituting huge changes in workflow and systems with AI to figure out what are those five whys, what’s the root cause of your bottlenecks, what’s preventing you from moving the needle ahead? And what you’re going to find is a lot of it comes down to execution gaps.

It’s the difference between knowing and doing.

You know, you have the information in front of you in a dashboard, but what are you doing about it? How soon are you doing it?

That’s where people start to see the value of AI. That’s where they start to get over their fear a little bit. Because you can train them how to use the tools to get the outcome and the impact of the outcome that moves their needle, their personal needle forward.

[00:25:23] Isaac Sacolick: Thanks. Joanne. Do me a favor. Before you have to drop off, do drop a link to your article around the five whys into the comments stream. Can you do that for us?

[00:25:32] Joanne Friedman: Sure. And I also have another one which is the seven A’s that every CIO and CTO should be looking at before they go down this road.

[00:25:41] Isaac Sacolick: Yeah, I think he’s.

[00:25:41] Joanne Friedman: A lot of it has to. Yeah, sorry. A lot of it has to do with outsourcing your judgment.

[00:25:47] Isaac Sacolick: We’re going to have another conversation around judgment. But those, the five whys and your seven A’s are good bridges. From this concept of let’s get people to use things to what I hope John and Joe and Eliz will start commenting on, which is like, how do we start talking about metrics that are not consumption metrics? They’re not sufficient. What should we be measuring early in the pilot or 30 days after a deployment that tells you your business outcomes will follow once we get adoption and scale. But John, let’s talk about adoption and let’s talk about metrics.

[00:26:26] John Patrick Luethe: Yeah, I agree that the adoption is the three things. There’s a change management side of it, which I think people have done a great job.

The one point I was going to add is that if you’re in a factory and you just go to one machine in the middle, you make that machine like 100 times faster. It’s not actually going to change the output of anything in the factory because all the other machines in the process are going at the same speed. And so that’s the, you know, the law of constraints. And so like if you speed up one part of the process, it doesn’t usually change things. And so that, that happens all over with AI.

And I think that the other, the next thing I was going to say is, is that when people are trying to use AI in organizations, the systems aren’t AI addressable. And so it’s, it’s like you go to try to do something and you want to use AI to make a process. A lot of the systems aren’t really at this time set up to talk to AI agents. And so over the next couple years you’ll see massive amounts of work for people to make agents so that they can talk to the applications that they want. And you’re starting to see this with the stuff in the industry. There’s mcp, right. And so almost all the applications you want to use are starting to advertise that they have MCP connections. Well, all your custom build applications, they don’t have MCP connections.

[00:27:41] Derrick Butts: Right.

[00:27:41] John Patrick Luethe: That’s something you’re gonna have to build in all your custom applications.

And so there’s, there’s so much stuff that, that, that in order for you to really use the benefit of AI. Yeah, there’s going to be a lot of human changes and a lot of system changes. And then the very last part is, is you go to run AI on a, on building an application or something. And it’s, it’s the Same things. You just, you just accelerated the, the type, the part of building the application, but you just, there’s, you slow down everything else, the requirements gathering. Well, sure, you can describe what you want and the application will go build it, but it’s probably going to kick back a ton of questions to you on exactly how you want to build it.

And then on the testing edit, it just expands the testing effort. So there’s a ton of methodology there and really, really exciting stuff. On the metrics, I was just going to touch on a few things. Everyone is talking about how to go faster with AI, but there’s a million metrics that matter. But there’s two that really matter to me, which is, if you’re going to go do something with AI, should you really be doing it? If you’re going to build something with AI, should you really be building that thing? So just ensuring that whatever people are doing, they’re doing things that are the highest value. There’s not enough conversation on that one. And then the last one is the quality.

If you’re doing something with AI, is it still high quality? And if it’s not, that’s a problem. And I, we’re seeing a lot of issues.

People are building applications with AI, and from what I’m seeing in the enterprise, the quality is going down, the code quality is going down.

The other thing is, is that the engineers actually don’t understand what’s being built anymore. So when you start having a problem, they’re going to have to troubleshoot somebody else’s code. And that’s, that’s going to be a real challenge. And so those are just some of the teasers. There’s 10 out there, but I wanted to give the other people a chance.

[00:29:24] Isaac Sacolick: John, I think you got one around measuring quality of AI’s output. I mean, it’s not intuitive, but you know, when people are telling you, hey, it’s doing something of value here, the value, you know, you can debate, you know, if you don’t have that articulated. But when you have people saying, hey, it’s doing a good job, then you could start asking, is it. What is it doing a good job at? And is it actually, you know, where is it actually impacting your time or your work? And it just leads to a lot of better questions.

[00:29:58] John Patrick Luethe: I got one more. So. So there’s this company and it’s, I won’t say the name, but they implemented AI into their ticketing system. So every ticket has to go through AI. You can’t call them anymore because when you call Them, they transcribe a ticket, their ticketing system. And so like when you have a major outage, you have to get through the AI to tell them that there is an outage. And I know, know people that use them and it took four days to get through to a human to actually say that there was a production outage. And that was the msp. And now, now they’re dealing with massive issues because they have to take all the AI out of their thing because it wasn’t classifying issues appropriately and there was no escalation path. So from a metrics perspective, it was 100% efficient at deflecting all the first line tickets. But on the other hand, they were, the customers were never able to get into the actual help that they needed. So 100% failed.

[00:30:49] Isaac Sacolick: Got it. John, thank you for being here. Joe, let’s, let’s take your comments and then I’ll take my break after that.

[00:30:55] Joe Puglisi: Okay, well, I’m going to take this in a different direction, Isaac. I was forbidden on the communications aspect,

[00:31:02] Isaac Sacolick: so I, I have to do that nowadays. Joe, you know, like, you know, this is a hundred ninety one programs. You know, we, we need to give our audience some more teeth.

[00:31:15] Joe Puglisi: Absolutely, absolutely. And everything we’ve been talking about up to this point, they’re all great points, brilliant observations. But what we haven’t talked about is what about the board and what about senior management?

It’s the CIO’s responsibility to not to say to people, do as I say and not as I do.

You have to be a leader.

In all the years as a cio, I never asked anyone to do anything I wasn’t willing to do myself.

So what we haven’t talked about is the need for the CIO first and foremost, and then members of management, including up through the members of the board, to explore AI and know what it can do, what it is and not what it isn’t.

We have a serious case of what we all used to call airline magazine syndrome, perhaps today as social media syndrome.

But the board and perhaps even the CIO might have some concept of what AI could do for the company.

But if you haven’t exercised it, if you haven’t touched it, seen it, felt it, played with it, you really don’t understand. And AI isn’t this monolithic tool that does everything you can think of.

It is, as we’ve talked about, lots of different kinds of tools, generative agents, you know, you’ve got to know what these things are. So my advice is start with yourself as the leader of this Initiative, understand enough about it to speak intelligently about it. Second, educate the other managers. That middle layer especially, it’s essential that they have an opportunity to understand it and play with it. And then, and only then can the board and the CIO and senior management begin to formulate that strategy. As Liz points out, that leads to expected outcomes that are realistic and achievable.

[00:33:26] Isaac Sacolick: Love it, Joe. I mean, you gave me my communications without using the word communications. But I think what you’re talking about here is more than just leading by example. I mean, this is something that’s impacting all of our jobs and all of our ways of delivering value.

And so what also comes out of this? I heard this from Jeetu Patel from Cisco just a couple days ago. He’s the president of Cisco and their head of product. Because it’s so much easier to do things, we need to be able to communicate what we are not doing.

And that’s one way to back into setting priorities and saying, look, we’re not going to focus on these five areas, we’re going to focus in these areas. Let’s talk about some of the things that you’ve experimented with we should learn from. I think those are all very valid ways of driving a dialogue with your staff. It’s very hard to ask those questions if you’ve never used the tools. I think that’s your point, Sandy. I’m going to bring you right back after the the break. Then we’ll go to Liz, Martin and Derek. Today we’re speaking about change management and adoption. Knowing that adoption is a step in the direction of delivering value that we not can, cannot always express what value and what our strategy are, strategies are. Until we experiment, the technologies and their capabilities are changing very, very rapidly. And so leaders, boards, managers and employees must work much more collaboratively together to get a sense of what are the tools doing. What can I do with them? Thank you for being here for this week’s coffee with digital trailblazers. We meet every week to talk about what’s impacting AI and transformation leaders in their organizations. Our discussion next week will be on AI infrastructure depth, strategies for capacity, resiliency and modernization.

We’re doing all this with AI and with data and is our infrastructure and our management and our security around it keeping up that? It will be our discussion next week on the 16th, on the 23rd, a leadership discussion. How do we finish Strong closing out the year and on the 30th, we’re going to have our debate about which agents belong as agentic and Fully autonomous. Which one should we design for human augmentation? And how do we orchestrate human in the middle as a way of building trust? Folks, I’ve left you a couple of banners here for things that I’m doing in the next few weeks.

If you are in Houston, I’ll be leading a roundtable that I’m calling the Breakfast with Digital Trailblazers. It’s about infrastructure leaders on AI’s hidden data debt. That’s on the 27th and that will be in Houston. You can see the URL there. I will paste it into the chat and then I will be doing, I’ll be a panelist on this discussion on the next chapter for ITSM.

That will be on October 22nd. And of course we’re going to be talking about AI and how that impacts how we service our employees in our IT organizations. Do visit these URLs so you can join if you’re in Houston on the 27th and everybody on that webinar on the 22nd.

Sandy, I’ll bring you back. Just give Everybody a quick 30 seconds on what you’re doing these days and then let’s talk about your metrics that you think are important to be considered when we’re talking about AI adoption.

[00:36:57] Sandy Mccarron: Sure. So I recently started working with Smush Parker, an NBA veteran, on a product called Virtus.

It’s basically an identity record for athletes starting with basketball. And we will be working on adding other sports. So it’s actually really cool to have a, a cool startup with a lot of distribution and a lot of really enthusiasm. And nobody knows this, but I actually did play basketball with my astounding height of 5:3.

I was a forward, but imagine that.

And then just to pivot back to our, our discussion, I, I think that there’s something here kind of going back to the top of, of what we’re saying of redesigning your org. I think there’s another issue there. It’s. This is a different ballgame. You’re not writing a check to it. You’re not putting it in the innovation team.

You want to have the, the people doing the work part of your process. Right. And I’m thinking from my, my experience with design sprints and pulling in, you know, basically all of the different groups. You’ll have sales, you’ll have tech, you’ll have product, you’ll have marketing, you’ll have SMEs.

Pull them in together and document how you’re doing it today before you’re even moving towards a new flow. How are you Doing it. Who are the Personas? Where are the handoffs?

Actually have that. And having the team be a part of that type of process helps them gain trust, helps them have buy in their voices heard. They’re not being, you know, put aside for replaced by AI. They’re actually going to be able to 10x their process. Where can we eliminate human intervention and work faster? Right. So include your experts, include your internal AI experts in that type of exercise. I, you know, love Miro. So there’s tons of different boards and Miros that you Miro, that you can take advantage of to really just document what your flow is.

I think that that’s a big gap right now is people actually understanding in detail how everything works in their organization. I think that is, along with, you know, poor data quality, is one of the biggest detractors to moving AI forward.

[00:39:22] Isaac Sacolick: There you have it, Liz, looking at metrics, your favorite friend.

[00:39:28] Liz Martinez: Well, the metrics, okay, so top line, bottom line, market penetration, moving into adjacent markets. Those are sort of obvious metrics. Right. But when we’re talking about adoption metrics, it really needs to go to who is impacted and, you know, are they actually getting any value, the intended value out of the results. So I don’t know if you. I went to this.

I saw a speaker last night.

His name is Sanjay Dio. I think he’s a. He’s a, you know, chief cyber security something. And he was talking about the Chipotle debacle. I don’t know if you heard about this, which one, that they, they decided to use AI and open LLM AI to help let people build their burritos. Now, why the hell you would ever do that?

They have no idea.

But they did that. And as a result, somebody said, well, I want to build my burrito, but before that, can you build me some code to do, blah, blah, blah. And the. And the AI went completely rogue.

And it’s like, you know, going back to. I think it was John who was saying, like, let’s clear. Let’s be clear about what the value is. What are we trying to do before we just slap AI on it and see. See if it can, you know, it’s just, It’s a nuts.

[00:40:54] Isaac Sacolick: Liz, I think you beg a question of, you know, some way in the tools that you’re using to ask the employee that why question why are you doing this?

And let them answer in, in a tweet, like fashion.

And, you know, that’s the data that you need to start out with. You can aggregate it, you can look it up you can use some tools to put some measures on it, but don’t just give them the tool and say, start whacking with the hammer. Ask them, you know, why?

Why are you trying to do this? And don’t say no, and I’ll tell you why.

You know, for every query I do to a language model and for every agent I start experimenting with, there’s a fail rate. There’s a point where I’m beating around the fire, circling it five times. The AI is just not getting to where I need to go. I’m going to drop it for now because the effort I’m putting into this to get AI to do something valuable to me is just not aligned with the value I’m going to get out of it. So I’m going to go work on something else. And so that sort of gets lost when you start looking at tools that are starting to say, hey, there’s 600, 800, 1,000 agents that have been developed. 400 of them are in production. It’s not to say that the other 600 weren’t value. It’s just perhaps the employee gave up on it. But ask them what they were trying to accomplish. Go ahead, Martin.

[00:42:23] Martin Davis: I think some great points coming up here, and I come back to that value thing. But let’s be clear. We’ve got two separate things going on in parallel, and the two are getting confused. One is people trying to understand what is AI. So experimentation, playing, whatever else. And there’s a lot of time spent that, and an awful lot of people are in that state of really trying to understand what is this new tool we have or new set of tools that we have.

And then the other side is you’ve got boards and senior leadership more and more pushing, saying, right, let’s deliver value from this, because we can see that there’s value from it. How do we get to the value? And things like that. And those two things are kind of not quite lining up because the tools are still trying to mature.

There’s a lot of people not fully understanding what the tools can do and what they can’t do. And that’s changing daily as well.

So we’re kind of in this constant state of flux where the boards and the senior leadership are trying to define, yeah, what’s the value we’re getting for it. And everybody’s busy playing and trying to understand what it is. And you come back to, you know, from a senior leadership standpoint, what are you trying to do? What value are you trying to derive? You know, are you trying to help them, the customers. Are you trying to help? Yeah, the bottom line. Are you trying to develop a cure for something, whatever is. What are you trying to do and how are you going to get there? But that’s very difficult when you’re on shifting sand and sometimes quicksand underneath that’s constantly changing and trying to figure out what it is.

[00:44:00] Isaac Sacolick: Hey, Martin, I’m going to put you on the spot here, right. I’m going to tell you, you’re a cio, you’ve gotten budget for experimentation.

You’re six months into the program, so not too early to really say, hey, here’s the ROI or here’s, you know, measurable impact on some of the areas of value. You’ve articulated them. You said, you know, we want employee satisfaction to go up by three points. We want our call centers to have higher deflection rates.

We want to take some defects, you know, some percent of our defects out of key processes that we’re doing.

We want to, you know, you know, we want to shorten the cycle it takes to hire employees by using AI in our hiring process. You’ve articulated value. It’s too early. But you’re presenting to the board in November and they want an update on how you’re performing. What are some of the things you’re going to talk about there? Because giving them an answer and saying, I’ve got 30% of the company using AI is a consumption metric. Yeah.

[00:45:08] Martin Davis: What does that matter exactly? Looking at specific projects. So, you know, let’s take the call center type thing. The customer service side of things, I would be looking at, okay, if we’ve been developing an agent, a bot that is going to help with dealing with customer inquiries. Have we been piloting? If we’ve been piloting, what are the early results from that pilot? And I’d be using, yeah, real world examples of that and saying, okay, yeah, this is where we’ve got to. This bot is doing pretty well. It’s able to cope with the majority of calls that come through, but there’s about 40% of the calls coming in that it can’t cope with. It doesn’t know what to do with at the moment. So we’re looking at extending its training, extending the knowledge base it has so it can start to answer more of those questions. And then when we get to a good degree of our confidence in it, we’re monitoring it, we’ll continue to monitor it. But, yeah, can we then open it up to a greater proportion of the calls coming in? So I’d be Looking at physically how we’re doing, I’d be looking at what we need to do to improve it further. And then what would it take to actually make it production ready?

[00:46:19] Isaac Sacolick: I think what you’re saying, there’s a subtle detail you’re missing in there, which is even though you’re still early, you have articulated where the value is coming from. You need to drill into your data.

Right. And get the stories that come out of that.

[00:46:36] Joe Puglisi: Right.

[00:46:36] Martin Davis: So I’m getting practical, I’m getting really into how and why are we going to use this and where we’re falling short.

[00:46:44] Isaac Sacolick: Exactly. I think that gets lost when we start talking about gross metrics. And what you’re really trying to do is get into, you know, what parts of what’s happening in the trenches are heading in the right direction, what parts are not. Go ahead, Derek.

[00:47:01] Derrick Butts: Yeah, I agree with what Mark was saying, but one of the things I’d also say is a company setting up realistic expectations for everything that he just mentioned, because a lot of times I find they have these expectations, we need to have this done by a certain time frame and those time frames are off because the business culture hasn’t fully adapted to what they’re trying to roll out. So one of the things I’d put in place in looking at this is make sure leadership’s aligned with around six things. As Martin mentioned, the business outcome, what problems are we trying to solve, looking at the workflow, what processes will actually change when, if it works. I look at accountability, who owns the AI enabled processes and its outcomes. I’d also make sure from a cyber and resilience point of view, what guardrails are put in place, what boundaries around the data, the security, the privacy, the human oversights, and the acceptable use. I would also look at the measurement, how we measure the behaviors, the business culture outcomes to tell us whether we’re doing this well, what needs to be modified and stop. And the last thing I would do is looking at what are the feedback loops to give us the true feedback of what’s working and what’s not and what risk has been impacted versus what’s not. I mean, if leadership can’t answer these questions, I think it makes it better for the initiative to roll out, but also get the business culture involved to help them solve it. It can’t be leadership pushing down. It has to be from the top down and the bottom up, where they meet in the middle, where they’re all working together to make this happen.

[00:48:21] Isaac Sacolick: Derek, I’ll tell you yesterday again at the Cisco Event.

Jitu Patel laid out six areas of investment Cisco is doing, but then he was very clear about what his non negotiable number one priority is.

And there’s a lot of comments here around communications, around leadership, you know, taking all the noise and saying this is my non negotiable number one priority and making that very clear with people. I think it just needs to be said when we know we need to give tools out there for people to try and learn from. We know learning isn’t necessarily the same thing as your POC or pilot. Right. I’m going to give time to my CIO to be able to learn some of these tools. But the way that CIO is learning or that CISO is learning isn’t necessarily a POC or a pilot. Don’t look at them the same way. Right. POC or pilot should have objectives. Folks, I’m going to go into my last question. You could comment on anything you want, but I’d love to play this little game. John. Liz.

I’ll bring Sandy back after that. Martin. We’re going to talk about change management. We’re going to roll back the clock a little bit because there’s still many organizations that are early in their journeys around this. So I want to talk about if you’re starting an AI change management program, you want to make sure that the money you’re putting in now around AI is going to start delivering the value you’ve got everybody on board with the idea of change is important to do up front. You’ve articulated your strategy and value. Okay, what would you put in place around change management before rolling out the tools, before saying what your pilots are or even defining it in any initiatives. Go ahead, John.

[00:50:13] John Patrick Luethe: Well, I think the first thing is, is that you, you really have to explain that why you’re doing this one. And I think it really starts with trying to get everyone in the org to understand that AI is not going away. And it’s a very transformative technology that’s like the Internet. It’s like electricity. That’s going to change every part of our life, basically. Right. And so I think you start with why. And then I think you, you, you really have to do real change management for things. You have to enable the people, you have to show the vision, you have to have, you know, change management activities, coaches on this one, tools, and, and you have to run it like a real program. And so that’s, that’s what I think. If you want to make a change in the org, you have to really explain to People. Why?

And then you have to, you have to really run it like a program. And, and if you don’t do that stuff, you’re just, you’re not going to get the right outcomes.

[00:51:03] Isaac Sacolick: Go ahead, Liz.

[00:51:04] Liz Martinez: Yeah, I hate to say it. It’s no different than any other rollout, just like, just like John said. But I, I think that the culture that you’re rolling anything out into is definitely going to impact your org change rollout. So, you know, if in fact you have a fearful culture of anything AI Just introducing it and getting people to try some things so that they’re not afraid of it is very valuable. I think you mentioned that at the beginning. But other than that, if you’re doing your AI in a, In a. As part of an initiative that’s aligned to your strategy, that has a business outcome that is focused on actually getting the biggest business outcome, AI is only the means.

So it’s not to some degree emphasizing that it’s AI and having people quote, unquote, you know, be more comfortable with it. That’s not the point.

Right. The point is to increase customer retention, increase the top line, go into new markets. Right. That’s, that’s the point. And that’s where people should be focusing and on the why. And that’ll help you in terms of your adoption, your communication, and making sure that you’re doing a lot of feedback.

The feedback loops are, I cannot be underemphasized. Making sure that people are comfortable with the, with any new initiative and measuring. On a scale of one to five, do you feel like this is actually helping you? On a scale of one to five, do you feel comfortable with it? On a scale to 1 to 5, do you know where to go, ask questions, blah, blah, blah. Those are very, very valuable data points that help you guide your communications. I know you don’t like using that word, but your training, your support, your mentoring, whatever that might be.

[00:52:55] Isaac Sacolick: I have another good contribution here from Wayne about incentivizing the behaviors you want people to be following through. Whether it’s around trust, whether it’s around empathy, there are people going to be in different situations.

Whether it’s about giving people time to learn, I think that’s all good areas of change management to focus on. I love Liz’s suggestion.

Build the feedback loops from day one. Go ahead, Martin.

[00:53:24] Martin Davis: I think there’s a lot to be said about change management here.

I think we have some issues related to how we go about change management generally and how change has changed and the speed of change is causing us Big issues.

If you think about, like the ad car model. One of the issues I think we have is in quite a few people, we have desire, as in desire to use AI, but without really the awareness or knowledge of how to use it.

And equally we have the. The inverse of that. Instead of desire, we have fear.

I was, yeah, really thinking here. We have a lot of people that are very fearful of it as well. So we have the negative going on too.

And I think all of those changes need to be very much defined. We need to look at how we go about things. And I’d start from the very beginning. First principles of change management.

What are we changing and why?

Then helping people to understand how they move through that change. So helping them to gain that more knowledge.

[00:54:35] Isaac Sacolick: And by.

[00:54:35] Martin Davis: Through that knowledge, they can then decide for themselves whether they desire it and how it’s really going to help and how they think they can actually benefit from it. And there’s going to take a lot of work and a lot of time through some of that. And then you’ve got to actually think about, well, how do you lock that into the organization? And Duane said incentivizing. But the other side to that as well is removing ways that you don’t want people doing things. So blocking off ways of doing things in undesirable ways, blocking off more manual ways of doing it. If you have a better AI tool that’s going to help you move through it. So a lot of words, but a lot of change management principles that date back to the 1950s and 1960s really still do apply, but just in different circumstances these days.

[00:55:23] Isaac Sacolick: I’m going to keep going around the room. We’re down to our last five minutes. Go ahead, Joe.

[00:55:29] Joe Puglisi: With all due respect to my colleague, Ms. Martinez, I do think that there’s a marked difference in change management with respect to AI.

It is all the usual things that we’ve talked about many times in the past and brought up today as well. But here I think we have to stress the point that the rules are going to change.

The rules are going to evolve through this process.

It’s not like the usual change management where we have an objective and we sort of know what the rules of the game are to get from point A to point B. Yes, it’s gradual adoption, it’s education, it’s experimentation. It’s all the things we’ve talked about, even communication.

But in this case, because the technology is so new, so different and changing, as was said earlier, so rapidly, I don’t think we have a set of Rules that will persist, they won’t be the same. Let me say it that way. The rules won’t be the same throughout the implementation. They will evolve just as the technology is evolving.

[00:56:39] Isaac Sacolick: I love that statement. I mean, and I probably would add one more statement to it, which is get used to feeling uncomfortable. Right. You know, this, this idea that if we can put effort into learning a new tool that, and align with business strategy, everything is going to fall out. Okay. After that, I don’t know if we can make that statement. What do you think, Joe?

[00:57:03] Joe Puglisi: I, I think you’re right. In, in, in terms of expectations. Right. The expectation here is we don’t know what to expect.

Right. We like to set expectations, but this thing moves and changes so rapidly and in, in ways that we can’t anticipate fully.

Yeah. You have to get used to being uncomfortable. So expect to be uncomfortable.

[00:57:29] Isaac Sacolick: Thank you, Joe. Sandy, I’m going to give you the last word for today. Thank you for joining us.

What are you saying to people around change when you’re walking in? You know, they’re just getting started with things and need help with employees who are fearful. And, and maybe if they’re not fearful, they’re just not aggressively pursuing learning and trying capabilities or even management is just not stepping up and saying, look, this is why we need to do things.

What are you putting at the front of your change management program?

[00:58:07] Joanne Friedman: Okay.

[00:58:08] Sandy Mccarron: Well, I would say, I’ve heard recently this, the saying adoption is not binary. It’s a gradient not to trash Liz and jump on the train. But it is very different in terms of change management because there’s a lot of feelings that are attached to AI right now. The constant news cycle of data centers are bad and there’s all these AIs coming for your job.

I think what I would do in any organization would start with a survey. Everybody has the tool to do this with Microsoft and you know, they’re embedded everywhere. Can do an anonymous survey, ask them their feelings about AI, what they think their, their level of expertise is, do they have recommended use cases.

There’s going to.

What you’ll find is where you’re at your baseline as a corporation or a company.

You’ll also start to pull the thread on who your early adopters and champions could be.

And you’ll also find if you see 60% of the people are responding like we need to fix our customer success team. AI is where we want to start, then, you know, you’ll have a really great starting point.

So I would start with a baseline survey and Maybe to Liz’s point, sorry, I feel like I don’t want to dump on her. Continue the survey so that you have a baseline and you can see the improvement and that it’s measurable. It’s not just, oh, I made more decks, it’s how have we made impact on the actual work that we’re doing.

[00:59:47] Isaac Sacolick: I love the idea of surveys. I love the idea, let all this

[00:59:50] Liz Martinez: go without me responding.

[00:59:52] Isaac Sacolick: Go ahead, Liz, you’ve earned it.

[00:59:55] Joanne Friedman: Go ahead.

[00:59:55] Liz Martinez: So it’s not about like ignoring the fact that it’s AI or ignoring the fact that people have different levels of comfort with AI, but if you’re focusing on how, how comfortable people are with AI, it is actually taking your eye off the ball. Point of the AI is to help you get some business objective and you need to focus on the business objective.

[01:00:21] Isaac Sacolick: So here’s how, here’s, here’s how I end this, Liz, because it’s going to, you know, I love the idea of surveys and I love the idea of creating feedback loops. I think those are two things that I would do very, very early on. I would add a third one, which is create a cadence of collaboration points, right?

[01:00:38] Derrick Butts: Yes.

[01:00:39] Isaac Sacolick: Turn this into an agile process and say, you know what, this group is going to meet every two weeks and the larger group is going to meet every six weeks. And all we’re going to talk about is share stories about how we’re using AI, what we’re trying to accomplish, why it was important, what did we learn from the exercise? I think the most important thing to learn about change management that’s different in AI is bring people together to discuss is something that management just needs to do. Without it, you’re just going to see data and numbers and tools. You’re just not going to get that storyline that says, hey, we have something going on here in this part of the organization.

How do we double down on it? Or how do we catch early that we have a few people in the organization that are either fearful or maybe using AI that’s not ready for prime team yet, or using AI in something that’s low margin, low value and be able to give them that feedback early. So create your collaboration points, open that dialogue up and good things will come. Folks, great conversation today. Thanks for all the comments on the LinkedIn Live. This will be up over the weekend. If you missed the part of it, including this dashboard, my research slide is already posted at drive.starcio.com/coffee on the 16th, we’ll be talking about AI infrastructure debt. We’ll be talking about how do we make sure that our systems aren’t preventing us from exploring our AI objectives and goals or becoming liability from a performance from a resiliency or security standpoint. We’re going to get into the technical weeds next week on the 16th. On the 23rd, we’ll be talking about finishing strong, how digital leaders close out the year. On the 30th, we’ll be having the debate around our agents and whether we’re going co pilot, autopilot, human augmentation. We’ll be going through all that discussion on the 30th. Folks, do sign up for the webinar that I’ll be on on October 22nd, the next chapter for ITSM. That’s at starcio.com/ai/itsm. And if you’re in the Houston area, I will be leading a breakfast with digital trailblazers on AI’s hidden data debt. And you can sign up for it at starcio.com/ai/houston-roundtable. Everybody, have a great weekend. And you know what? We might have to think about experimenting here. We’ll talk about this in the green room after. How do we collect everybody’s great stories about how they’re using AI into some kind of form? Maybe we maybe that’s a group project we have to think about. Everybody, have a great weekend. Thank you for joining this week’s coffee with digital trailblazers.