Summary

This week’s “Coffee with Digital Trailblazers” episode focused on evaluating AI technologies with special guest Maribel Lopez, a top industry analyst. The discussion centered on three key areas: value, risk, and ecosystem considerations when selecting AI platforms. Maribel emphasized the importance of starting with clear business objectives and measurable use cases rather than getting caught up in the vast number of available AI agents and tools. The panelists discussed how organizations should assess vendor trustworthiness, governance capabilities, and the potential for platform lock-in, with Joanne highlighting the need for governed autonomy and coordination across multiple AI agents from different vendors. The conversation also covered the challenges of balancing short-term value generation with long-term strategic planning, as well as the risks associated with data privacy, model drift, and system dependencies. Participants agreed that organizations need to focus on their specific business goals rather than being overwhelmed by the numerous AI agents available in the market, with Maribel noting that many current vendors have simply added AI capabilities to their existing offerings rather than providing truly transformative solutions.

Speakers

• Host – Isaac Sacolick
• Guest speaker – Maribel Lopez, Enterprise AI Industry Analyst | Agentic AI · AI Infrastructure · Governance · Customer Experience | Keynote Speaker · Podcast Host · Forbes Contributor
• Digital Trailblazers – Derrick Butts, Martin Davis, Joanne Friedman, John Patrick Luethe, Liz Martinez, Heather May, Joseph Puglisi, Elena Putilina

Discussion

1. What 2-3 capabilities matter the most when evaluating an AI platform’s capabilities — and everything else is secondary — what makes your list, and what’s overrated?
2. Cost savings are easy, but not transformational. “Business value” sometimes gets to fuzzy definitions and debatable metrics. What should Digital Trailblazers use to evaluate AI-enabling technologies and their ecosystems that captures today vs future benefits?
3. There’s an encyclopedia of technology and vendor risk considerations. What’s new, different, or has elevated importance when evaluating AI technologies and their ecosystems?

Research

White Board

Transcript

[00:00:01] Speaker A: Welcome everyone to this week’s coffee with digital Trailblazers. I am super excited for this episode.

This has been a long, long time coming.

Reuniting with my good friend, top analyst Maribel Lopez, who is joining us as a special guest. I actually have a full house of all of our advisors here to talk about evaluating technologies in the AI era.

Value, risk and ecosystem. I picked those three areas to focus on because my feeling is that the tools that we’ve been using to evaluate technologies don’t really apply very well when we’re looking at AI capabilities and they don’t work very well when we’re selecting AI platforms and technologies and looking at how do we differentiate and create opportunity with AI in our organization. So that is going to be our conversation today. I’m going to jump right in because like I said, I have a full house. Everybody is here, I think, and Maribel Lopez, who has been a longtime analyst and a friend, is here to help us ground our conversation on the reality and the future of AI and how we think about selecting technologies and. And in my research today, just to get us grounded, I broke this down into two areas, value and risk. To get us started with, we’re looking for platforms that will help us grow and help us improve our organizations. 19% say AI initiatives have met or exceeded their goals. 18% say fewer than 33% of AI use cases meet defined metrics. This is coming from CIO.

I love this data point from our friends at Futurum.

AI ROI measurements of top line revenue growth and bottom line Profitability doubled to 21.7% and productivity gains collapsed 5.8%. We’re much more interested in growth than just finding productivity. For those of you who want a board perspective, you can go to Protiviti’s research around this talks about the top areas that company that boards want to see from AI and then this discussion around AI. Native companies are operating at different scales with some achieving 300,000 to $450,000 per employee. Some of the risks, data to privacy, regulatory and governance. We’re more concerned about that than the workforce impact.

So for those of us who want to talk about risks, I’m, you know, that’s a softball to my friends Liz and Martin to talk about change management.

There’s a very interesting set of questions in Kai Wehner’s research. How much do you trust your vendor AI and how much are you prepared to get locked in?

One of you can pick up that topic. The enterprise’s leverage is highest before signing and lowest after two years. Of integration. Some data points around that from Arjun Jaghi. Very interested to see now for me, and I’m just going to sort of throw this out there, right? When I look at potential partnerships and vendors and solution providers, I am looking for some way of measuring ecosystem. And so that’s what you see underneath there under Star cio, the seven different ways I look to see whether this company is healthy.

I look at their depths of partnerships, whether their APIs are well used.

How easy is it to get data out of the system?

Is it a unique skill set or can I hire partners and people for that skill set? I go to their conferences and look for community sentiment. In other words, I’m looking for raving fans.

And when I look at their website, I want to make sure that their leaders are accessible on social and that they have relevant use cases published that mean something for the industry that I’m working in. So just some of the areas that I look for around ecosystem value and risks. Again, I want to welcome Maribel Lopez to the floor.

Maribel, you, like me, invested a lot of time in the second quarter looking at what the different solution providers are putting out there from an AI perspective. And so this first question is really for you. What are the two or three capabilities that are mattering the most when evaluating AI platforms? What wowed you and what is really primary on your list when you’re looking at solution providers? Maribel, welcome to the floor.

[00:04:56] Speaker B: Thank you. Okay, it’s a big question, right? But let me see if I can unpack it a little bit because I. Well, maybe I’ll start this way and say, I think two things are happening in this space and these are not unusual things, but they’re more difficult to deal with now. And the first thing is you get like highly specialized AI providers that will just provide one piece of the stack. Maybe they’re talking to you about agent identity, maybe they’re talking to you about observability.

Then you’ve got the other side of the stack which says, hey, when you think about moving towards agentic AI, we’re looking much more at this concept of really a new software development life cycle. It’s like an AI development life cycle. You’ve got to deploy your AI, you’ve got to manage it, you’ve got to test it, you have to have identity and security and observability and governance around it. And I’ve seen that if you look at, for example, the hyperscalers, I think we were at the same conference and somehow didn’t get to meet up in New York for aws. But that was a good example of the other side of that where it’s like, okay, here are all these pieces that you’re going to need and we’re trying to pull them together for you in an integrated platform, which has a lot of value but also has the difficulty of, well then you’re bought into someone else’s platform and you have to connect that to your software as a service providers and see if those things work together back on your ecosystem. Comment.

So the things I look for is I look for how many pieces of this platform can I get from one vendor. I look at how easy is it to connect into other pieces that I might want. And it seems like right now the answer to everything is just like, well, we’ve got an MCP server and I’m not sure that that dog really hunts, but okay, it’s better than you not having an MPC MCP server. So we, we’ve got that.

And then, you know, so that gets to, to your ecosystem partnering. I don’t want to call it open because I’m, I’m not sure that we really have as much openness as people claim we have.

But just also the sense of there is a cadence to what I’m giving you and you can see new features roll out in two months, six months, because this is such a dynamic environment and we are dealing with the unknown unknowns at this point. So it seems like every month you’ve got your main partner providing you with new functionality because that’s the pace of innovation. I don’t know if that exactly got to your question, Isaac, but the two thing, three things I’m looking for are how many pieces of the platform you can provide, how easy is it for you to pull in the other things that are my key, you know, software applications or other parts of the stack.

And then what’s your real cadence of innovation versus the PowerPoint slides that you sell?

[00:07:46] Speaker A: Mervel I love how you unpack this, especially because you know, you’re presenting it in through the lens of the buyer, right? And so I probably have multi cloud and hearing from AWS and Azure and probably Google about what they’re offering.

And my dev teams are hearing about all the vibe coding platforms that they want to go rush and do and build everything under the sun.

And then I have an ERP and I have a CRM, whoever the vendors are that I already have and they’re exploding with the number of agents that they’re offering. They’re giving you development studios, they’re giving you some variation of a way to orchestrate workflows across all this stuff with their integration promises.

It’s, it’s like, you know, which door do you want to open? So maybe unpack that sort of landscape and say, you know, how do you advise leaders about how to not get trapped in an analysis paralysis, but also not experiment everywhere and get nothing into production and value?

[00:09:04] Speaker B: If I look back at how this has rolled out, you know, there’s, you know, in some ways where we’re all still humans, with the exception of the new AI agents, and we still have the same challenges that we’ve had before.

We just now have the AI lens on it. So what do I mean by that when I talk to people about unpacking things? Like, hey, listen, you’ve got one or two or three really key projects, objectives, momentums, whatever you want to call it in your business that you are trying to accomplish this year.

And where we started with AI was like, well, let’s just do a proof of concept and see what AI can do. And it’s very science experiment. And I think that was the nature of it, say a year, year and a half ago.

And I think now if you talk to an organization, the first thing you have to do is I’ve got people that are saying, I need to prove in some value on something I deploy in a quarter or two quarters. And that’s a very different mindset in terms of how you deploy things, because you have to start with, what’s the use case? Can this benefit the use case? How am I going to metric this use case? And I think everybody gets hung up on, did I, did I save money? I mean, there are other things that could happen. Yes, you could accelerate a process. Yes, you could save money. It could be that you get to new product introductions faster. You know, there’s, there’s other ways to metric things. But I think where I tell people to start now is it’s just not about the tool. It’s like we have to figure out what we’re trying to solve, how we’re going to measure that, who’s going to own it. Like, how do you know what success looks like with your AI deployment?

And what success are you looking for in the short term versus the long term?

So if, you know, Isaac, you remember back in the day when we were looking at Iot and we spent all this time picking an IoT platform and we connected all this stuff to the IoT platform so things could talk to each other. And that took like years at this point, and we got to the end of it, then your manager looked at you and said, okay, so we got everything connected. What are we doing? And, like, nobody had an answer.

So I think we’re trying to move beyond that. So the ch. The only challenge with this is that if you’re not doing a unified strategy, you end up with all these different projects all over the company, and that ends up being expensive and it doesn’t scale and all this other stuff. So that’s why I think when we try to rally around, say, what are the top three things that are going to move the needle for your business, either in terms of driving revenue or saving money, you start to get to a better answer around what tooling goes there? And then you have to be willing to pivot. You know, this is. This is not a market where you say, oh, I think AI innovation is going to stop two months from now. So I’m just going to wait until it slows down and then I’ll pick my platforms. You have to be able to say, I’m comfortable with change and I will pivot if I need to, and I’ll stop there because it gets rambly after that.

[00:12:00] Speaker A: But no, you’re hitting all the points. And I love the IoT story because it’s an IoT story. It’s a mobile development, mobile first strategy, a the big data strategy. You put all that stuff in our data centers. It’s like, who’s customer number one for this thing?

I got a bunch of hands raised who probably have their own versions of this story. Let’s go around the room, Derek. We’re talking about capabilities that matter around AI platforms and then just your thoughts about how you evaluate technologies in the AI era. Hey, Derek.

[00:12:36] Speaker C: Good morning and welcome. Mary Bella. Yeah, Mary Bella, a good point. Talking about unified strategy. When I look at this from unified resilience strategy, the questions that come to mind, especially with this AI stuff, is looking at can I trust the company, how well do I govern, how well do the. Does the business create or do they have hidden risk? And I look at how they play in the marketplace to see what exposure they’ve had and other customers they’re looking at. So I’m looking at the security of the integration to my business flow. I’ve already got a roadmap, got a plan, how well will they adapt to that? What capabilities and tools do they have? They’ll meet my current requirements today, but also scale to tomorrow. I’m looking at how they’re going to connect it to my data, will I overexpose sensitive information. What are the role based access that can configured to work with this issues? I’ve got a set of compliance needs, do they conform to those and fit that model? I also look at the way they’re modern monitoring their systems across the board and say if they’re monitoring, what compliance are they working with? Is it nist, artificial intelligence, is it Monitor atlas? All these things come into play when I’m looking at AI governance, but the governance piece of it also is I’m looking at how well are they doing it themselves, the monitoring the governance, the systems behaviors that they have, how well are they monitoring use of data? I’m looking at the outputs that can be generated and how those can be reused.

I’m also looking at decision making processes. As far as the evidence of the system, do they have AI capabilities to monitor things like drift of the data or misuse or hallucinations? So as I’m looking at these AI tools, what AI mitigation strategies have they already integrated into that? And most of all, if I have this ecosystem, this platform that I’m looking at, I understand my current data, I understand my current risk, is it going to be adaptable? Can I work with this as a third party coming in? Well, all the other systems that I need for it to talk and tie into, will it be able to do that quickly or does it have to be customized? I’m looking at also how my business culture would probably integrate into this and the evaluation, the training, all these things come into play.

When I look at this across the board I’m really looking at does it have the things that I need to fit what I’m trying to do today and will it scale to where I see myself based on how I’m looking to use artificial intelligence tomorrow?

[00:14:46] Speaker A: You know Derek, you led off with the question can I, can I trust the company over a two week period of time now that OpenAI and now Anthropic has broken out of jail and hacked into.

I mean that I don’t want to go there. I mean maybe that’s another topic for

[00:15:03] Speaker C: us, but that’s reality.

[00:15:05] Speaker A: I mean, look, I think the big thing here is, you know, this notion of vendor viability is really important, you know, and that doesn’t necessarily mean don’t work with small vendors or don’t work with startups. It means dig deep and understand roadmaps and understand, you know, where the levers are on their costs.

Because we all know that we’re paying 10 cents on the dollar for the AI models as John likes to remind us that, you know, so at some point, you know, the pricing is going to go up and so we have to match who we partner with with value. Let’s go to Joanne. Joanne, welcome to the floor.

[00:15:50] Speaker D: Good morning to you all.

I’m trying to figure out where to start. This is like so packed.

But let me start not only the platform capabilities, but also I want to say a quick word, and my quick word is, please stop squaring the models.

Because if you don’t stop squaring the models, you’re going to be behind the eight ball. Because it’s what happens after the model decides that you really need to worry about my three things for capabilities from vendors and having, you know, two lenses being applied here, one looking at them and judging them, and as an analyst and the other as a builder.

I’m going to mix my metaphors here. So I would say my first three things are governed execution, not can it answer if it’s an agent, but can it act?

Can you see why it acted? Can you see where it stops or where it reverses? Well before damage would land.

Somebody has to be accountable. That’s human on the hook. And they have to be accountable not only to the build of the system, but to the corporation and the policies. The second thing would be coordination at a fleet level. You know, everybody’s talking about, oh, well, we solved an agent problem.

That’s true. A loop can complete, a harness can be used. But 40 agents from four vendors on three different continents, that’s a bigger problem.

Unless you have an arbiter in the mix, you’re always going to have a problem. And that goes to something that Maribel said, which is about, you know, a multitude of providers, SAS providers, how things are going to connect, etc. This is where you get into the choreography of agents as opposed to the orchestration of pipelines. And that’s a big problem that people haven’t quite solved yet. Some of us sort of have.

And what is overrated is benchmarking. Single agent reliability. You know, Derek talks about trusting the company, as does Isaac and several others. It’s fine to be skeptical of the company, but what I think is really important is the reliability of the agents and how they’re crafted.

That gets into some of the tooling. But I’d say from a trailblazer perspective, autonomy, maybe a dime a dozen. It’s governed autonomy. It’s how can you take something that’s designed to be autonomous, fit it into the culture of your organization?

That’s not Quite there yet. And so I would say a trusted path and a phased approach would be two other points that I would add.

[00:18:37] Speaker A: Thank you, Joanne. I think Maribel had wanted to chime in on governance. Go ahead, Maribel.

[00:18:42] Speaker B: Just one quick comment. So nothing stops a conversation in a cocktail party like the word governance.

Everybody agrees they want to do it, but then no one does it. We’ve looked at a lot of different research from different companies, including my own. Only about 50% of the companies actually have a solid governance strategy. And then they’re going into AI, so they don’t know what they don’t know. And the third thing I’d really like to say is every single technology vendor is going to tell you that they have a great security identity and governance stack that they’re going to give you. And that all breaks the minute you start crossing stacks. And I’m not sure I believe it.

So what we’re really running into is we’re running into a scenario where we have extremely immature technology that is very difficult to secure with immature governance practices of most organizations. This is a disaster waiting to happen. Particularly as you put the word autonomous into it. It’s like what could possibly go wrong with that? So I would say you have to really proceed with caution and be very thoughtful about how secure that is.

[00:19:52] Speaker D: If I can. I just want to comment back. I totally agree with what you’re saying. Some of us have taken this two levels of depth that are for a different day in a different audience. But the governance issue is one that is not just based in what may be existing corporate policy. It’s the new what do you need to now look at from the point of view of AI governance, which is an entirely different beast. So sure, policy, security, identity, you know, the registration of agents and what their allowed to do goes to agency. And I think that’s going to be another topic in governance that’s going to rear its slightly ugly head over the next six months to a year. And it is a tough problem to solve.

I think that’s part of the reason why Human in the loop, human on the loop and human on the hook, which is the person who is accountable for it all. That whole pathway has to be part of what would be AI governance. So I agree with you that it’s a level of maturity in the tooling.

[00:20:58] Speaker A: Yeah, let’s, let’s agree to that. We need another topic on AI governance in itself and I think the perspective is that, you know, whereas the beginning of this year most companies were in just trying to get you know their first agents into production. Now I’m hearing some. The leaders getting into hundreds and thousands of agents and production. And like Maribel has suggested, every vendor has to have a governance story because that’s exactly what they want. They want lock in on their platform for you to run and connect lots of agents. And so they have to have a story around that. We’re going to have to cover that in another episode because it’s just too big of a topic. Martin Garvanen isn’t important. What is important?

[00:21:47] Speaker E: Oh, I was going to go for a whole, whole raft of things here. It just kind of.

By the way, I came across an interesting one the other day. This is kind of a little bit on the governance topic, but I’ll move off quickly.

One of the software vendors I was looking at, and this was a piece of financial software, you can trust our AI agents because they’re ISO certified. And I hadn’t come across that one yet, but I’ll throw that as a kind of a.

Kind of one for the governance discussion about whether that actually means anything or not.

I was going to go into the, the whole thing around change management, so you threw me a kind of a softball there, Isaac. But I was also going to go into. And you guys are going to think I’m a broken record, but why are we doing it?

What is the value from doing it and how does the, the platform align to what we’re trying to do? Because I think we, we very quickly get the shiny object syndrome going on, especially with AI being, you know, the potential for making massive difference to most companies.

So we start to lose track of our common sense saying, it’s a tool. What are we doing and why are we doing it and what value do we hope to get? And that earlier conversation around getting to the end of the pilot with IoT or whatever else, or getting the end of the implementation with IoT and then saying, okay, now what.

[00:23:11] Speaker F: Yeah.

[00:23:11] Speaker E: Is a clear example of that very thing. And I went through some of the IoT stuff in the manufacturing space, as did Joanne, and I think it’s, it’s very much a case of you start with what’s the problem you’re trying to solve and how you’re trying to solve it. Obviously, with AI, you’ve got to think bigger, you’ve got to think wider, you’ve got to think differently, but it’s still very much a case of you’re trying to run a business here. How do you improve that business, how do you make more money, how do you actually deliver More value to your customers, all of those aspects. So I would very much go into that focus and then get onto the change management side of things is how can that vendor help you introduce these new tools? How can that change in the organization be done in such a way as to be accepted by the organization? So I’d be asking the vendor some very serious questions about how they go about implementing and what experience. And I want to talk to reference customers that have actually done it. And how does that AI platform help you actually introduce itself into the organization? Because that’s something from a change management perspective we’re not used to, we’re not used to tools that are actually capable of helping you to implement themselves.

[00:24:27] Speaker A: Martin, I will just say this.

You know, my early first review of any platform starts with ease of use and onboarding and speed to value.

If I can’t get there fast enough, easy enough, it’s not even worth looking at anything else.

There’s just too many choices, too many categories, too many vendors. There’s a lot more work that has to go into looking at governance and scaling, adoption and you know, looking at versatility, short and long term value that if it’s, if ease of use isn’t there. Not even talking to you. Go ahead. Joe.

[00:25:07] Speaker G: Well, as usual, Maribel covered a lot and, and stole my thunder with, you know, and Martin as well with, with, you know, know what you want to use it for. That was kind of my lead point. I, I want, I’m going to give you three bullet points that I think distill down in my mind to three key questions that as management I would be asking of my IT department.

The first is is it fit for purpose?

Assume we know what we want to do with it. Is this the right tool or the best tool to achieve whatever that target is that we’ve set as, as a goal. So fit for purpose is kind of bullet point one.

Number two is what’s the approximate investment necessary? What’s the cost structure here? How much do I have to spend before I start to see some results, measurable results that I can put up against that, that investment. And maybe there’s not a return on investment in dollars and cents, but there’s learning or you achieve some other objective. But it’s measurable, it’s got to be measurable.

So the cost structure is the second and the third. I’m stealing from you, Isaac, because you prompted a thought in my head. We kind of, we need a sort of a yelp for AI, right? We someplace we can go or multiple places that we can go to get the vibe around the solution that we’re entertaining.

[00:26:30] Speaker A: What is the buzz?

[00:26:31] Speaker G: What do people think of it? Have people had good results or horror stories?

Where can I find that kind of information? That’s what I’d be looking for. So those are my three bullet points, fit for purpose, the cost structure, the investment needed, and some kind of a Yelp feedback mechanism that tells me I’m picking a winner or avoiding a loser.

[00:26:54] Speaker A: Joe, there are a lot of review sites out there right now and I think they’re worth looking at. I don’t think they’re worth making an investment decision off of what I see in there.

I do think the best way to get vibe is to go to the conferences and to go meet customers and talk to people who are actually using them, hopefully without the solution provider directly in the room.

And yeah, there are some platforms when you go that you will see fanatics, people who just love the platforms and that’s generally a healthy sign.

[00:27:33] Speaker G: Well, I said because, you know, I have an extensive network, I wouldn’t necessarily have to go to a conference, but I could just reach out to, I don’t know, 20 or 30 people and get feedback pretty easily.

[00:27:46] Speaker A: Joe is silently plugging networking, which is actually how Maribel and I and Joe met and pretty much everybody here for that matter, who’s here as a speaker, he’s silently plugging, joining the Society of Information Management where he and I and others like Liz and Derek, we regularly meet there here in the Tri State area. Elena, I’m going to give you the quick last call on this question. Then we’re going to jump into the other two and Maribel, prepare a short statement about what you do for a living for everybody to hear and we’ll get to that next flow. Elena?

[00:28:27] Speaker H: All right, I’ll be short. So a lot of good things have been said said on assessing the platforms at what matters. What I want to bring up is that just about every current vendor that we work with now slapped AI or very thoughtfully introduced into their offering.

And especially when it comes to your commercial team, we are all using it and we need guidance from the tech team on what’s viable, what’s not. And all the current vendors need to be assessed for viability just as much. And therefore all the current commercial counterparts that are using it need to be managed and explained to what are the implications, risks, etc. So my favorite word cross functionally. So this issue is handled cross functionally and we don’t have any issues within the existing stack that could introduce quite a bit of risk as well.

[00:29:25] Speaker A: Thank you, Elena. Folks, you have joined this week’s coffee with digital Trailblazers.

Our sessions cover AI, digital transformation and leadership topics. We’ve been meeting here for nearly four years. Over 180 episodes that you can get more information on@drive.starcio.com Coffee Previous episodes are there. In case you missed one, these research slides are available there and the dashboards that we create are available there. So do use that as your resource. Our special guest today is Maribel Lopez, one of the top AI industry analysts. My opinion, the top.

Maribel, just do a quick intro. What do you do for a living? Who are you? Just let my audience know what you

[00:30:14] Speaker B: do and do so well, I’m a technology industry analyst. I don’t do stocks. I actually look at technology and try to figure out how different technologies can help enterprises deliver business value as the shortest way of doing it. So does it work? Would it work for you? What kind of use cases are beneficial?

That’s what I did.

[00:30:34] Speaker A: Super. That’s a good segue to my second question.

We talk about new technology. The easy thing is finding cost savings, but they’re not transformational.

And then we get into this debate. Can we actually demonstrate roi? I’ve done some writing around this and I’m like, if you’re looking for ROI in your initial investment, you know, draw that nice hockey stick and talk about fuzzy numbers. You probably are spending too much effort trying to get down to the dollar and cents. And then when we get down to business value, sometimes those metrics get fuzzy and hard to define. So you’ve already covered this a little bit. You know, what should digital trailblazers use to evaluate AI enabling technologies and their ecosystem that compare both short term benefits and establish opportunities for the long term? So this is really a question about short versus long term thinking, Maribel.

[00:31:31] Speaker B: I mean this is not an easy question, Isaac.

All right.

[00:31:34] Speaker A: There’s no softballs here.

[00:31:38] Speaker G: Okay.

[00:31:38] Speaker B: So when I talk to people about AI investments is really expensive and they can go off the rails very quickly.

[00:31:45] Speaker D: Right?

[00:31:45] Speaker B: So I think we’ve all had the token maxing days and those are behind us even for technology vendors. So we’ve moved out of that. But there are low hanging fruit use cases that I talk to organizations about. Almost every company you walk into probably has some kind of contact center application where they could either deflect password resets or just take some calls off the table or route them more efficiently. So that’s an easier one.

There are it operations use cases to Keep the network running to make the network more efficient. Those are good.

We’re starting to see some more interesting ones that aren’t low hanging fruit but are different in areas like HR where the ability to go through and screen more resumes and to actually use AI to interview candidates is a really interesting new way of doing things in organizations. So there’s some top lines and some bottom line. But I always say to somebody, if you, if you’re going into it today cold, which not that many people are going into it cold now, but if you are going into it cold, you need to be able to demonstrate some kind of value. It’s not always money. Sometimes it’s process improvement, sometimes it’s money, sometimes it’s the ability to do something you couldn’t do before. Like so for example, in the contact center people would measure 10% of the calls maybe at most. Now they can do 100% of call transcript on their contact center and figure out problems I didn’t even know existed. Agents that are doing well. So there’s definitely opportunities for how you measure the technology. But I think it gets back to what I first said around the business case and the font’s really small for me to read. What was the second question?

[00:33:26] Speaker A: Oh, we’re just comparing short versus longer term benefits. You gave us the short view and I agree with some of your use cases, especially customer support.

[00:33:37] Speaker B: The longer term view is a little more difficult because I think each time we go into technology the way we go into technologies, we typically replicate what we had in the past.

And I think what, and we’ve always talked about low code, no code, having citizen developers and all. So none of those concepts are really new.

But I do think we have better technology to actually make it so that more people that have an idea about the business process could actually transform that business process.

We could get to the point where the really difficult to use applications like you mentioned ERP and the like moving forward.

I think the biggest long term strategy that we never talk about in AI because we’re so obsessed with the tooling of AI is your data is just not right.

It’s bad data.

So the first thing I’d recommend anybody do is to just get their data house in order so that you have the truth and you can operate from the truth. And that doesn’t even necessarily have anything to do with the AI tools themselves.

So I’ll leave it there because I see hands raised and maybe there’s an opportunity for somebody else to chime in on that.

[00:34:49] Speaker A: No, it’s Maribel, it’s all good stuff and I agree with your customer support is ripe with good use cases. I heard a story yesterday, I won’t say who it was, but a big four accounting advisory firm using AI to conduct their stakeholder interviews. And I was like, would I actually outsource that? I don’t know if I would do that, but it is very interesting. And you know, that’s where experimenting is coming from. That’s where looking at departmental use cases is coming from. And then I agree with you, we are not at the point where we’re reinventing yet. And I’m going to be talking about a lot more about reinvention in the future. But let’s go to John. John, welcome to the floor. Sorry Maribel.

[00:35:37] Speaker I: Hey Isaac, thank you for having me on. And just going back for one second. I think actually if you’re doing interviews and you can record them with AI, it is a really helpful thing being able to have that transcript and going back, it can be a super helpful thing. But would I have them conduct the interview? No.

Back to the real point I want to make.

If we’re trying to use AI for the long term, I think the things that are most important is how do you handle upgrades to the model? How do you handle upgrades to new AI technology that comes out and then really the most important thing, and I was in an industrial IoT company, industrial analytics company is what does the breakup plan look like if you decide that you’re going to leave this company? What does it look like for you? The thing I really found with industrial IoT was it was so fundamental to the, to the company that people really didn’t want to have third party companies being so critical and, and a part, like a non replaceable part of the company that a lot of them just took it in house. And so I think anytime that you’re looking at some one of these AI platforms, you really have to understand what does it look like if I decide I’m not going to be doing business with this person anymore. And like how would I put a different, either a different model in as a minor change or maybe an upgrade or how would I do like a complete divorce? And what are the divorce proceedings look like?

[00:36:59] Speaker A: That’s a really tough one, John. I mean like AI agents from one platform, one model are not going to be easily interchangeable from another platform, another model. Right. And even in the, even in the more basic, even in the most basic stuff. Right. You’re just going to get very different answers for a lot of different reasons.

But you know, we’ve also learned all too often that we don’t budget for life cycle, we don’t have plans for testing.

And between those two things, it’s the number one reason we end up with tech debt. And it will be the number one reason we end up with AI debt. Maribel, you’re going to jump in on something.

[00:37:42] Speaker B: I was just going to say one of the reasons why I’m reluctant to do like the long term use of AI is exactly what we were just talking about is there’s just a tremendous amount of change.

So it’s hard to even imagine what platforms are going to look like. And to give you a use case of that, if you go back to, I believe it was early April, Harrison Chase from LangChain put out this missive on what a harness was. And before then nobody was even talking about harnesses and that was April. And now every time you turn around somebody’s talking about harness engineering. So the, the rate of change that we’re looking at is very significant. And I think that when I was talking about probably the best thing you can do for AI right now is try to build in how you can pivot.

Because that discussion about like the models changing, the agent platforms changing, you might love, you know your Google cloud instances today and how they do with models. You might want to be with AWS like six months from now you might decide all the models you’ve used are too expensive and you need to figure out how to get cheaper models. You also have to balance like what’s in the cloud versus what’s on prem. I have never seen so many organizations upgrading on PREM infrastructure to support certain AI workloads. It’s a big change and we haven’t even talked about sovereignty. Depending on where you are in the globe, there’s a whole discussion around what AI technologies to use, where do they reside, where does that data reside and how do you deal with that. So that’s why I’m reluctant to go with a really long term view on it. And I think the idea is that you do your best as an organization to plan to be as flexible as you can within your constraints. Right. This is not an easy thing to do.

[00:39:30] Speaker A: No. I’m really curious what Liz has to say about this because here we are under pressure to move fast.

We gotta bring use cases to production in one to two quarters. The board is knocking down our door. We figured out some areas of value because we there’s enough use cases out there to find some of the low hanging fruit and here we are talking about governance and, you know, exit strategy and life cycle management and do I trust the vendor and all the things that can go wrong? Liz, how do you sort through all this?

[00:40:03] Speaker D: Good Lord.

[00:40:04] Speaker F: So I will, I will lead with my usual. Governance is not a four letter word.

Governance actually is there to support the business in making sure it’s getting what it needs.

That’s in all aspects. That includes, you know, making sure you’re not giving away the farm and getting security leaks and et cetera, et cetera. So I’m, I’m a big proponent of governance and I’m a big proponent of pragmatic governance and common sense.

Listen, AI is a fabulous tool, but we need to make sure that we are checking our common sense, bringing it with us. You know, what are we actually trying to do? What are the business problems we’re trying to solve? And if we focus on that, everything else will sort of fall into place. You can’t just say, oh, it’s a new, shiny new tool and I don’t know what I’m doing, so let’s go play with it and throw a bunch of money at it. And for no reason, because it’s cool. I mean, that’s like buying a Lamborghini because it’s cool. That’s just nuts. You got to think about, do I need to take my kids to school? How often do I go grocery shopping? Do I have access to transportation, you know, public transportation?

What am I actually trying to accomplish and how will this tool get me there? Okay, now having said all that, whenever you buy a car, you actually also think about how much are the maintenance costs, what’s are the insurance, what’s going to happen, how long is this car going to last me, and what’s going to happen when I have to swap it out? What if it becomes too expensive? What if I got the Lamborghini and I actually have to, you know, swap it out because it’s just gotten too expensive and I want to go to a Prius. How hard is that going to be for me to do?

Obviously, a car is a standalone, but.

But when we start thinking about, you know, having all this infrastructure and all this investment in the learning in the tools, how difficult is that going to be to actually switch over to an entire new ecosystem? I mean, on the one hand it sounds incredibly difficult. On the other hand, it actually could be a lot easier the second time around because you actually thought through what it’s doing for you, what it’s not doing for you, and what you’ve been getting from it and what you’re looking to get going forward.

So anyway, that’s my three or four cents worth.

[00:42:35] Speaker A: So in summary, you’re saying focusing on governance is probably a good thing once you know what your targeted goals are and your objectives and how you’re going to measure it. But Martin, I mean like I did an article around this. I won’t name the solution provider. In 2015, at their conference they had 40 agents available.

This year at their conference they had over 200 of them. That’s a lot of squirrels to go chasing after. And you know, their sales reps are coming after our end users and departments and say go and try this one.

So you know, how do you make sense of how do we pick the areas to focus on and you know, what are the capabilities and how do we separate short and long term value out of it?

[00:43:22] Speaker E: I was going to take this to an even darker level.

[00:43:25] Speaker A: Oh no.

[00:43:27] Speaker E: So one area that I was going to be exploring a little bit recently was AI agents to assist lawyers.

And just to think this through. And there’s some very large companies out there that actually have AI agents trained with their body of knowledge that are actually.

[00:43:46] Speaker B: Yeah.

[00:43:46] Speaker E: Provide quotable legal precedents and things like this. So yeah, you can actually reduce your external council costs and those are pretty high. Yeah. Per hour rates by using these AI agents to help you review contracts and things like this.

So let’s just play this one out a little bit.

So you’re starting to use this for all of your contracts. You’re having it do legal reviews for you. Okay. You’ve got a lawyer who’s going to verify the stuff and things like that. But it’s taking a lot of your external costs out. It’s starting to build this internal knowledge of all of your contracts.

Now let’s take what we were just talking about in terms of does this, these agents actually survive longer term? You started to rely on it. Your body of knowledge has built up, your body of legal knowledge, your body of contract knowledge has built up within these agents.

What’s going to happen to that information?

What’s going to happen to your ability to carry on operating if that company then ceases to exist or gets bought by somebody else? Yeah, there’s kind of. You can actually go some very dark ways on some of this in terms of the short versus longer term aspects where you’ve got some very sensitive data.

[00:45:10] Speaker A: Martin, you did you cut off there or that was the end of your thought? No, I, I was, I was, I

[00:45:14] Speaker E: was doing a pause Pause for effect

[00:45:16] Speaker A: and same pause for effect.

The dark, dark, you know, look here, here, here’s a chain of thought there because I had the same question with the big four, you know, accounting firm person last night, right? So now you take something like legal or accounting and you’re using AI and you’re doing a lot more with less and guess what happens? I start demanding a cost savings out of my service providers and that’s happened.

It has a long history in it, right?

They find efficiencies under your contract, you expect to see a discount over time and you negotiate around things like that. And so what are the service providers have to do? They have to go back to the well and redevelop their value proposition back to their clients and otherwise they’re just going to have deflationary pricing. Right. And we’re all going to demand cheaper or no spend on my legal because it’s going to be all AI generated or you know, I’m going to find really trusted worded BE advisors that are providing value add AI plus human augmentation. Joanne, take us out of a dark space. I’m going to have Derek coming up next. We’re going to talk about more risks again. But Joanne, take us out of a dark place.

Why should we be doing this?

[00:46:37] Speaker D: We should be doing. Well, first of all, I, I would, I was going to talk about the, the longer term but first of all, years now is months or even weeks and we have to, you know, definitely take that into consideration because of the pace of change. But you know, this goes back to the first question and even to the second in terms of long term,

[00:46:59] Speaker F: when

[00:46:59] Speaker D: you look at things, I look at it and maybe from a skewed perspective, but if I wear the analyst hat for a while, I look at what is, what is the purpose, what is the use case. Yes, all of those normal things and how easy it is for a person to use. But this is where we get into things like business context. And to Martin’s point about law, it only works if you’re in the same jurisdiction.

It only works if you’re looking at the same types of contracts for the U.S. i think it’s almost laughable when I see some of the agents that people are creating because every state has different laws. So that’s times 50 in Canada. Well, you only have one that’s different. That’s Quebec Napoleonic law versus common law.

But country by country and territory by territory, agency is becoming a big deal.

Sovereignty is becoming a big deal and those have to be taken into consideration. Can you be model agnostic. Yes, but this goes to part of the reason that I say you have to own your own AI. The only context that really counts is your own context for your own business.

The models have nothing to do with it. They’re a tool and they, they’re being commoditized like a tool. How much governance you put around them is, is a different question.

But from the dark place to the light place you actually can create value and look at metrics like how often is the yield from the agent correct?

Like get brownie points for the right answer as opposed to the long twisted, you know, verbiage that some of the frontier models actually give you when all you want is is this correct or is this incorrect?

So from a long term perspective you want to look at stuff like that on the life cycle side. The other point that I would make is what’s becoming newer and differentiated and I’m kind of segueing here a little bit is the difference is for long term is not just the yield, it’s the compounding. Can you use these agents for more than one task? And I know fit for purpose is an issue but there are so many business processes and I think Martin will agree with me here, particularly in manufacturing that you have to take the upstream and the downstream into consideration. With every business process the same would hold true I think to this to customer relationship.

You’re not just looking at what is the value you can bring to the customer. It’s what is the value you can bring to their value chain just as much as the individual customer. If it’s consumer it’s me and my friends because I will recommend to my friends or we will discuss amongst us. So you have to look at the broader case as well. And what I see happening with a lot of the use cases is they’re very narrow and they’re not taking either an upstream supply chain or a downstream customers customer into account for the long term. And what’s becoming newer is how do you govern that then and how do you look at the risk that’s being coming to the forefront? Because at the outset the old risk was a wrong number on a wrong on a dashboard. Now the new risk is the wrong action in a physical world.

Well okay, that that turns out to be far more impactful because it could involve human safety. So we have to start expanding the mindset around all of this as well.

[00:50:37] Speaker A: I captured that Joanne, for you plan big beyond today’s AI agent narrow use cases.

That’s part of the reason just going shopping agent by agent, even when one does a reasonably accurate and delivers value is missing the point. And you can feel this yourself, right? If you start using AI, the very first thing you start doing is taking things you used to do before and now you’re doing it a little bit faster and smarter. And then when you expand yourself, you start doing things that you couldn’t do before, right? And you start doing things that are more interesting that you never had time to do before. And so now when you start boiling that up at an organization level, the organization can start thinking about doing things that it couldn’t do before. Derek, what are the risks with that? Wait. Well, we got to go around the room here. We got nine minutes. Sorry, Joanne.

[00:51:27] Speaker D: Okay, don’t worry.

[00:51:28] Speaker A: Go ahead. Derek, we need to talk about one thing here. We did talk a little bit about risks. There’s an encyclopedia of them. So I want to pick the ones from you pick your brains, Derek, or CISO on board, which are the ones that matter for AI right now. So the ones for AI, I mean

[00:51:44] Speaker C: looking at traditional risk versus AI risk, but it’s totally different focus. The older risks still prevail, but the AI risks are even more, more important. When you look at the data exposure, the things that come and the hallucinations and the data leakage, all those things now affect the baseline AI, affects the way the interacts with your services. Also the third party plugins, the agents integration, the way they expand those risks are huge and they’re real. The other risk of looking at a lack of explainability. When something happens, how do you know why it happened, how it was going to be repeated, the model drift, the reliance, all these things come into play. But the problem is when we look at this, when people installed these services, AR tools, they only were short sounded because they were looking at what it can do for immediate problems, manual problems. They weren’t looking at the long term effect in the business value. That was really important because they weren’t looking at the risk associated with trying to achieve those. So I also look at the ecosystem, the model providers and the other things in my ecosystem. If I’m building resilience model, everything that I put in that system has to have a resilience element to it. It’s an intentional purchase, it’s an intentional mindset. But it’s also looking at how I’m going to work with all my providers, all my Data Services, my APIs, my integration, all those at a level of complexity and risk. And, and I think people are short sighted when they look at trying to implement these systems in this new air, they don’t take all that into account and assume because they’re a big conglomerate that they already have all this figured out. They do not. You still have to do your due diligence and fall behind and say, does it really apply to what I’m trying to do? So really looking at this to go deeper, I would say, do I own the model that I’m working with? Do I own the risks associated with that? Do I understand the risk and how will this affect me down the road? Human oversight is still required to make this happen. You still have to have human in the loops to look at the outputs, decision and override systems, whatever. We still have to achieve confidence before we can achieve the.

What’s the word for me? For confidence before we achieve.

I’m sorry, Value.

Thank you.

But yeah, those are things we need to figure out what’s going wrong, how to handle it and make those things work in our perspective.

[00:53:52] Speaker A: Six minutes, folks. I got four hands raised and I want to hear last from Maribel. Elena, go ahead. Anything you want to comment on?

[00:53:59] Speaker H: Yes. So I want to build on what Derek was saying and dimensionalize the word value. And that means, and Liz brought it up briefly, what is the business trying to do that it couldn’t do before?

What kind of problems is it trying to do? Right, so stay close. My favorite word, cross functional with the business. Because it’s the business that would know these questions. It would also know requirements for federated data, etc. So that would allow you to understand risk versus value equation. Is it worth it to do it and assume the risk?

[00:54:39] Speaker A: Thank you for joining, Elena. Liz, what you got? Liz, you’re on mute.

[00:54:45] Speaker F: I think it’s funny that.

Thank you, Elena. Because this is really not about picking agents. I mean, the idea that there’s 200 agents at the conference and that they’re each going to be running you down. Try me. Try me. It’s just insane. Honestly, you got to focus on your business direction. What is, what is your. What are you trying to do with your business? What kind of margin you’re trying to capture? What kind of market are you trying to capture? What kind of new revenue streams are you trying to generate?

I mean, there’s always, you know, bottom line considerations as well. But I’m talking top line, mind. Let’s talk about where you’re trying to go strategically with your business. And if you can but fixate on those goals, then see what agents are appropriate to actually help you get there.

I can use another Analogy. I’m making amazing dinner for my friends and I think about what kind of experience I want to have. Then I go pick the recipes, then I go grocery shopping.

[00:55:43] Speaker A: Liz, let me, let me set the stage. When we talk about 200 agents and where we’re seeing them, they are largely in back office with the exception of customer support.

We are not seeing a whole lot of grow the business agents just yet. That’s just not where they are right now. And so when we talk about blue sky planning, it’s about, look, our back office is going to get smarter and more efficient and more data driven and more AI enabled. Now what? Go ahead, John.

[00:56:15] Speaker I: Yeah, I just, I think about this, I think the biggest, the biggest risk that we’re dealing with is right now is that we’re dealing with fast moving technology. We’re not paying the full price for it. And since it’s so new and it’s so good at discovering information that may not be accessible to people without AI, that I just, I think we want to have a partner that’s going to be really, really working with us and that if things don’t go south it’s, there’s, there’s a clean way to get out. And so that’s, I just, when I look at this, you know, what, what, what kind of company do you really want to be with? Are they really going to be here and are they really going to be really good partners to work with?

[00:56:55] Speaker A: These are really good points, John. I mean you’re really getting at the heart of we establish trust. Thank you, John.

[00:57:01] Speaker C: Yeah.

[00:57:02] Speaker G: Joe, I got two quick bullet points for you. The first is I’m concerned about lock in.

What happens when I have a potentially better solution or an alternative solution or this one stops working for me, how, how tied in them am I and I want to deal with that up front, not, not when the time comes that I want to move. The second is a big concern, dependency.

What happens when this thing fails? Is that going to take my business down? Have I planned for that? We don’t, we don’t always think about the dark side, but dependency becomes the second big concern for me.

[00:57:40] Speaker A: Go ahead, Liz.

[00:57:45] Speaker F: I’m not fast enough quick on the draw. The key here is that when we’re, we’re focusing these back office agents or focusing our energies on leveraging the agents in back office.

It frees up our best resource. Our best resources are our people, they know our business. It actually frees up their creativity to focus on top line.

[00:58:07] Speaker A: You know, there’s some numbers around that, Liz. That it’s not freeing us up, it’s giving us more to do. We’ll have to cover that at a later coffee hour though. Joanne and then Joanne and then Maribel.

[00:58:19] Speaker D: I’ll be very quick.

The, you know, to, to the point that was made about lock in. I think that’s one of the key considerations even now at the earliest point in time is if the provider is not model agnostic or not writing back information from agents in a way that is portable.

Be dismissive because change is not only constant, it’s accelerating.

It’s accelerating even more than you realize, not only in the frontier models but in the vendors capability to deliver that. You have the frontier models now being embedded into so many other products. As those change, you’re going to get locked in whether you realize it or not if you’re not careful. So flexibility, adaptability, beat observability and explainability 10 to 1.

[00:59:15] Speaker A: Thank you Joanne and all my speakers. Maribel, give us your last thoughts on this topic.

[00:59:21] Speaker B: The cool stuff has been said. I did have a discussion with, well, there was a discussion with young brands and they made an interesting point about scale. Whether we want to call it agents or skills or what have you. Just the concept of, you know, you’re, you’re in baseball, you’re creating a throwing agent and it could do many things like the reusability.

Right now everything’s a snowflake. Can we get to a point where we’re using AI to build reusable skills that can be used across the business? I think that’s an interesting point.

[00:59:53] Speaker A: Thank you, Maribel. And I want to thank Keith for this lasting comment. The dark side is always with us. The biggest problem with risk management is the lack of imagination of how bad things could go. And I will counter that and say our biggest problem is not using our imagination on how much we have to transform and reinvent.

We’re still in a place where we’re reshaping business with AI. There’s a lot of questions. There’s also a lot of opportunities, short term opportunities. Maribel signaled some of those earlier today.

And if you missed any part of this week’s coffee with digital trailblazers or want to listen to a previous episode, do Visit us@drive.star cio.com Coffee and you will be able to get access to previous episodes. Folks, I have the entire menu on the upper right hand side of the whiteboard of our upcoming topics. Next week, by audience demand, we’ll be talking about NETworking in the AI era.

We’ll be looking at job seekers, those who are prospecting, every reason why we are looking to network and how to do it. Today on the 14th, very interesting topic. Beyond legacy processes, how do we engineer the high velocity Enterprise on the 21st?

Derek’s going to be the star if he can make it Shadow AI at work, real risks, practical guardrails and hidden innovation. I don’t think Derek can make it that week, so we may have to move that one. And the 28th we’ll be talking about SaaS, cloud and AI contracts where technologies lose leverage. This is Joe’s comments and questions about how do we avoid lock in and a whole lot more, folks. The links to all four of those are available at drive.starcio.com coffee. You can click on them to join.

You can also use the Add to Calendar button so it shows up on your Google or Outlook calendar and you don’t miss an episode. Every one of these is also recorded in case you missed it. And I produce almost every episode on both Apple, Amazon and Spotify for you to listen to. Folks, Summertime we’re all happy because we’re ready to go out and have a great weekend. Thanks for joining. I’ll see you here next week.