What should an AI architecture mean today? If you’re developing AI agents and enabling agentic AI, can you just connect AI models, enterprise data, and workflows?

Not so fast, say the experts I consulted with on the important AI architecture rules that leaders need to excel now and in the long term.

Architects are among the key leaders in digital transformation in the AI era. They connect business value to robust, scalable solutions. Top architects avoid building point solutions that become the next generation’s technical debt.

AI architecture rules build upon many DevOps, data governance, and resiliency best practices. That shouldn’t surprise anyone, because AI agents are essentially genAI-enabled applications that connect to enterprise data sources. The more autonomous, agentic AI requires robust automation and orchestration capabilities. Essentially, genAI requires taking architecture best practices and scaling them, all the while the technology and AI models continue to evolve.   

Defining your organization’s AI architecture  

The challenge many architects face in defining an AI architecture is unraveling the mess. Every enterprise has its complexity of applications, siloed enterprise data sources, the agentic capabilities being added by many SaaS platforms, and the underlying data debt.

This article builds upon my previous work on defining a framework for making architectural decisions, which includes five questions to debate solutions. Experts shared their rules and principles for AI architecture, which enable faster and safer experimentation leading to scalable solutions.  

Below are ten AI architecture rules and principles to consider.

1. Take an incremental approach to applying AI agents

“What’s most important is that companies get started with AI, and for many, that means starting small,” says Rob Scudiere, CTO at Verint. Taking one specific process or micro-workflow and transforming it with agentic AI can serve as a proof of concept and a learning opportunity ahead of a broader rollout. Companies don’t need to transform every aspect of their operations with AI all at once. They can start with smaller-scale projects to deliver tangible business outcomes now, while also laying the foundation for longer-term results and ROI.”

My take: In rethinking the architect’s role for agile transformation, I suggested that architects take on agile roles, such as product manager over technical debt and delivery leader for extendable platforms. An agile best practice is to seek business-meaningful MVPs rather than targeting all-encompassing architectures and grand scheme workflow reinvention.

2. Avoid lock-in with a single AI model

“Organizations have a narrow window to get their agentic architecture and governance right and avoid having to unwind years of technical debt years from now,” says Florian Douetteau, CEO of Dataiku. “They must refrain from being lured into building agents in disparate, incompatible frameworks in their rush to deploy them. Instead, they must maintain long-term model optionality; now is not the time to get locked in with a single model provider. This requires a standards-based, modular approach to creating and controlling agents.”

My take: While data fabrics help centralize access to integrated data sources, the LLM mesh addresses an emerging need to enable architects to connect and control access to different LLM models. Architects in enterprises with regulated data sources and large-scale AI programs should review both.

3. Provide context rather than rigid requirements

“Avoid common anti-patterns when designing AI systems because traditional software development processes focused on strict specifications do not fully apply,” says Nirmal Mukhi, VP and head of engineering at ASAPP. “AI performs well with ambiguity, so it is more effective to provide context, tools, and clear instructions about what not to do. Testing should rely on a cycle of simulate, evaluate, and repeat rather than static test cases.”

Architects and CTOs take note. While developing AI agents may be an assembly process of integrating LLMs with workflows, the testing process shouldn’t be trivialized.  

“Another mistake is expecting zero errors, and unlike traditional systems, AI will occasionally produce mistakes,” adds Mukhi. “Designing for mitigation, setting clear user expectations, and building tolerance for rare errors are essential for success.”

My take: Investing in a test-driven approach to developing and validating AI agents yields longer-term benefits than trying to nail down absolute requirements.

4. Establish AI and data governance principles before developing AI agents

“The rapid deployment of AI agents without robust data controls can create systemic risks, including hallucinated decisions from misaligned data, brittle automations, and compliance exposure from opaque data flows,” says Srujan Akula, CEO of Modern. These risks stem from fragmented, unowned data. Connecting agents to governed, reusable data products—each with clear semantics, ownership, and lineage—ensures that AI acts on trustworthy, auditable inputs. This architecture is essential for scaling AI safely and sustainably across the enterprise.”

My take: Architects should partner with the chief data officer on developing a defense and offense AI governance strategy. This strategy should include developing an AI vision statement, building customer trust, defining data non-negotiables, and establishing ModelOps.

5. Control the information flow between AI models and APIs

“Agentic AI is forcing enterprises to rethink how systems communicate because connecting AI directly to production APIs creates an unsustainable mess of security vulnerabilities and technical debt,” says Matt DeBergalis, CTO, Apollo GraphQL. “What’s needed isn’t just MCP, but a deliberate orchestration layer between AI and services, with GraphQL as the ideal foundation, that enforces governance while enabling both to evolve independently at their respective paces.”

My take: Point-to-point integrations were problematic long before the advent of AI agents and other generative AI capabilities. Architects should define requirements and evaluate approaches that simplify addressing integration debt while also establishing a standard for AI integrations.

6. Build resilient and scalable architectures from the get-go

“The Cambrian explosion of Agentic AI we’re all bracing for may soon be followed by a mass extinction, specifically the applications that weren’t architected for ultra-resilience and scale,” says Andrew Marshall, VP of product marketing at YugabyteDB. “The vector databases that many companies use to spin up new AI are simply not built to accommodate massive scale and throughput while also providing an ultra-resilient experience for end users. Distributed, multi-model databases were built to accommodate these requirements out of the box, so the agentic AI built on top of them will be enterprise-ready from day one.”

Marshall explains further and says, “Data, specifically an organization’s data, will be the key to successful Agentic AI applications. LLMs alone are futile—it’s your business data and proprietary knowledge, paired with GenAI engines, that deliver value to users. These RAG (Retrieval-Augmented Generation) applications require the same cloud-native resilience and scale as traditional applications.”

My take: Scalability and resiliency requirements must be defined for each business and its specific use cases. Otherwise, development teams will make their own assumptions; some will target scale beyond reason, while others will ignore key non-functional requirements. Don’t assume that a solution doesn’t scale – define the requirements, validate existing architectures, and then look to evolve or pivot when required.

7. Define architectural non-negotiables

Kurt Muehmel, head of AI strategy at Dataiku, shares these four architecture non-negotiables for GenAI:

• Your AI must connect directly to production enterprise data, not copies or samples.
• It must work seamlessly with your existing analytics and ML models, rather than replacing them.
• It must plug into the systems that run your business—your CRM, ERP, and operational tools.
• It must have governance built in from the outset to ensure quality, safety, and cost control.

“These aren’t nice-to-haves—without them, you’re just playing with AI, not transforming your business,” says Muehmel.

My take: Refer to my previous articles on data governance non-negotiables and DevSecOps non-negotiables. Muehmel has a good starter list for your AI architecture non-negotiables.

8. Develop configurable, modular, and swappable architecture patterns

“For defining the architecture for the GenAI era, a core principle is ensuring flexibility and composability,” says Simon Margolis, associate CTO of AI & ML at SADA. “We need to move beyond monolithic structures and embrace modular designs that enable AI agents and services to be easily swapped and reconfigured. The architecture requires a strong emphasis on well-defined interfaces and robust API management as non-negotiables. This highlights the importance of open, common protocols, such as MCP and A2A.”

My take: Are AI agents in SaaS platforms monolithic structures, or will MCP, A2A, and other agent-to-agent protocols enable interoperability? Architects need to review and develop their strategies or risk extending SaaS sprawl into AI agent chaos.

9. Explain in concrete terms what guardrails mean and their importance

“AI agents can be powerful force multipliers—but rushed deployments without guardrails might turn them into liabilities,” says Sanjeev Vohra, chief technology and innovation officer at Genpact. “Organizations risk runaway costs, data leaks, and agents corrupting core systems. To avoid turning innovation into damage control, teams must embed clear boundaries into agentic architectures—defining what agents can decide, and what they must never touch. This ‘autonomy with accountability’ approach isn’t optional—it’s the only path to scale responsibly and stay audit-ready when things go wrong.”

My take: Guardrails is a less ambiguous term than governance, but still needs concrete definitions and explanation. Architects should partner with legal, risk management, data governance, and the CISO to set requirements that can be easily implemented across AI models and agents.

10. Define principles for reliability and guide agile team adoption

Mukhi at ASAPP shares these core principles for building reliable GenAI systems:

• Problem decomposition – Break problems into tasks and solve them via a plan; structured framing improves outcomes.
• Guardrails – Prevent and detect hallucinations or ungrounded responses with layered safeguards at runtime and post-output, ideally with explainability.
• Simulation & evaluation – Enable testing at scale by simulating diverse inputs and evaluating outputs for quality and relevance.
• Continuous monitoring – Use LLMs to assess and report system performance in real-time, enabling self-awareness and improvement loops.

My take: Reliability, like resiliency, guardrails, and scalability, needs concrete definition for each business and use case. I highly recommend an emphasis on testing and monitoring .

My last statement sums up my most important recommendation on AI architecture: Emphasis on testing and monitoring is highly recommended.