Some organizations select technologies largely based on vendor pitches, only to find a myriad of integration issues and customization requirements.
Others perform exhaustive searches, soliciting all stakeholders’ “non-negotiable requirements” and creating elaborate spreadsheets, aiming to quantify the technology that satisfies more of the most essential requirements.
These two extreme approaches rarely worked in traditional tech evaluations. In the AI era, they can be even more problematic.
Buy and contract too quickly, and you risk AI debt and AI cost debt. Apply too much thinking, and considering everyone’s priorities might box you into solutions for yesterday’s challenges, not future opportunities.
We discussed the challenge of evaluating technologies in the AI Era at last week’s Coffee With Digital Trailblazers. I broke the selection down to three key reviews: value, risk, and ecosystem.
“This is not a market where you say, oh, I think AI innovation is going to stop two months from now, so I’ll wait until it slows down and then pick my platforms,” said Maribel Lopez, our guest advisor on the episode, the founder and principal analyst of Lopez Research. “You have to be comfortable with change and willing to pivot.”
Simplify the value and risk criteria
One of my top workshops, 10 Steps for Technology and AI Selection, provides a process around the three themes of selection: value, risk, and ecosystem.
Value can be dubious. You don’t have the data to project ROI during technology selections. Qualifying business value, while very important, easily falls into the trap of fuzzy definitions and debatable metrics. “It’s not always money. Sometimes it’s process improvement, and sometimes it’s the ability to do something you couldn’t do before,” said Lopez.
And the problem with measuring risks during technology selection is that there’s an encyclopedia of criteria coming from CISO, legal, finance, and other risk management professionals. Derrick Butts, enterprise AI cyber-resilience and risk advisor, and founder of Continuums Strategy, said, “The older risks still prevail, but the AI risks are even more important — data exposure, hallucinations, data leakage, lack of explainability, model drift.”
John Patrick Luethe, a leading speaker at the Coffee With Digital Trailblazers and owner of Comfort Keepers Seattle, raised the one risk factor many organizations leave out. “You really have to understand what it looks like if you decide not to do business with this company anymore. How would you complete the divorce and what would the divorce proceedings look like?”
Ecosystem is the differentiator most important in the AI era
I led off the episode with my weekly research slide and strong opinions. First, I focus on time to value — if a platform is too hard to learn, use, or deploy to production, then it’s a nonstarter. No other criteria matter if the march to the startline is a hike before the marathon.
Then, understanding the solution provider’s and the platform’s ecosystem is paramount to selecting technologies that deliver short- and long-term value. This rings more true today, as AI capabilities evolve on a 2-6-month cadence.
I shared seven criteria to review around a technology provider and solution’s ecosystem. Here’s how I think about ecosystem evaluation, as a CIO/CTO for nearly 20 years, and now advising many of them.
1. Partnership breadth — is there one that will work for my organization
I will often engage the solution provider’s professional services during POCs. But using professional services beyond then can get expensive, while relying on my internal team to learn all the platform’s best practices and nuances is unrealistic. I seek out partners as part of the evaluation process, and evaluate whether they have enough to select from that know my industry, use cases, innovation opportunities, and risk profile – and are of the size that my organization can afford.
2. Data portability — ease of analytics and exit
What does avoiding “lock-in” really mean? At the very least, I want to make sure that I can get my data out, in a structured way, when (not if) required. But equally important to me is that I need this data for downstream uses. I can’t expect the solution provider to fulfill all the organization’s reporting and dashboarding requirements. Plus, whatever data is in the platform is likely needed to train ML models and by AI agents running inside and outside of it. Closed access to the underlying data is a nonstarter.
3. Integrations, APIs, MCPs — the context layer openness test
Beyond data, I want to connect the platform to complex workflows, AI agent orchestrations, and enable developer access via APIs. My pre-AI criteria included evaluating whether platforms like Zapier, Workato, and Tray.io had the triggers and actions available to build no-code integrations. Today, I’m reviewing whether the platform connects to data fabrics, if their AI agents can connect to external context layers, and what their MCP server capabilities look like.
4. Skill set availability — can you learn it? Staff it?
How do you grow internal adoption of a platform beyond the early enthusiasts? When I look to develop proficient users and internal platform experts, what I am really seeking are learning paths and recruiting options. One of the reasons I loved Atlassian’s tools beginning in the pre-AI days was their rich documentation and how Google searches with my questions returned the best documentation pages to review. Today, skill set availability can be evaluated by prompting your organization’s sanctioned LLM with technical questions, researching skill-based learning programs, and quantifying the talent pool available on job boards.
5. Community sentiment — fans vs. skeptics
One reason I go to a solution provider’s conference is to qualitatively experience whether customers are raving fans or sitting in the back rows with arms folded. Tableau, Quickbase, and Adobe are my high bar for raving fans. I also seek out skeptical CIOs and CISOs to determine whether their issues are concerns, biases, or not relevant to adoption based on best practices. There’s more to evaluate at conferences: the partner/solution ecosystem on the showroom floor, learning/certification paths through the solution provider’s university programs, and whether customers are truly excited about the new capabilities announced.
6. Leadership accessibility — is the brand alive?
Technology brands need spokespeople – those that engage with customers, partners, analysts, and the media beyond their product and features. The ones that fully understand industry challenges and opportunities, who can answer your questions – or network you to the right people who do, and the ones that demonstrate the solution provider’s brand, mission, and values. When reviewing a technology solution provider, it’s a red flag if you can’t easily identify these people. My high bar for this criterion is Appian, which has a leadership team led by Matt Calkins who have been together for over 25 years.
7. Relevant case studies — the credibility read
Case studies help identify the extent to which there are companies like yours using the platform successfully and willing to share their story. But go one step further and reach out to them, not to validate, but to find out what they did that made them successful, and how to avoid the speed bumps they encountered.
Is your tech selection process analysis paralysis, or are you speeding too fast and breaking through the guardrails? I’m here to help.
Leave a Reply