Where Does Enterprise AI Get Its Answers? Architecture Considerations for Procurement Teams

When organisations buy enterprise AI, they often evaluate features, pricing, and security controls without examining where the system actually gets its answers. That question has direct implications for cost, governance, auditability, scalability, and vendor lock-in. Enterprise AI systems commonly generate answers from a combination of model knowledge, retrieved organisational content, structured business systems, and explicit business rules. Understanding the distinction helps procurement teams assess what is actually being purchased.

Why the Source of Answers Is a Procurement Concern, Not Just an IT One

There is a tendency to treat architecture as an engineering concern that procurement does not engage with. That instinct tends to be costly.

The architecture materially shapes the commercial deal. A system drawing primarily on model knowledge often introduces usage-based costs that scale with activity. A system retrieving from an organisational document corpus adds indexing and corpus maintenance costs that scale with data volume. A system integrating with operational business systems adds connector and maintenance costs that scale with integration depth. A system built on structured relationship models adds domain modelling and governance costs that scale with domain complexity. These cost structures are not directly interchangeable. The architecture a vendor recommends commonly reflects what their platform does well, which may not align with what the organisation needs.

The architecture also shapes what is auditable, what is governable, and what can be explained when something goes wrong. Procurement teams who skip this question commonly end up with contracts that constrain governance options that have not yet been thought through. The connection to broader enterprise AI total cost of ownership is often direct: the source of answers is one of the largest TCO drivers, and it is rarely modelled at procurement stage.

What follows is a procurement-grade overview of the four common ways enterprise AI systems source their answers, what each commits the organisation to, and where each tends to fail.

Source 1: Model Knowledge

The simplest and most commonly procured configuration. The system is a large language model accessed through an enterprise interface. Prompts are entered by users, and outputs are generated from the model's training and any context the user supplies in the prompt. The model has no access to organisational systems or documents beyond what users include directly.

What it does well. Generic tasks where the model's training is sufficient and the cost of being approximately right is low. Drafting, summarising, brainstorming, generic question answering, and tasks where the user can verify the output quickly and discard bad ones. Deployment is fast, integration is light, and per-user cost can be predictable under seat-based licensing.

What it does badly. Anything that depends on organisational knowledge the model does not have. The model has no knowledge of the organisation's products, pricing, customers, contracts, or internal processes unless that context is provided in the prompt. Outputs that look authoritative on internal topics are often less reliable than they appear, because the model has been trained on the public internet and has no access to the organisation's data.

The commercial implication. A deployment drawing solely on model knowledge is procurement of generic capability, not organisational intelligence. The use cases it serves are real and worth serving, but they are typically productivity uses rather than operational uses. Where procurement is being asked to fund a deployment that touches operational processes, customer-facing decisions, or workflows that depend on internal data, model knowledge alone is rarely sufficient.

The failure mode to watch for. Vendors offering an enterprise plan on top of a pure model and describing it as if it accesses organisational data. It does not, unless retrieval or integration has been added. A useful line of enquiry covers: where does the model get information about the organisation, and what happens when a user asks something the model has no basis to answer?

Source 2: Organisational Documents and Content

Retrieval-augmented generation (RAG) is the approach most enterprise AI platforms deploy when they claim to "use your data." A retrieval system, typically a vector index built over a corpus of organisational documents, is connected to the language model. When a user submits a prompt, relevant documents are retrieved from the index and provided to the model as context. The model generates an output grounded in that context.

What it does well. Question answering, drafting, and summarisation grounded in a defined organisational corpus. Policies, product documentation, customer support knowledge bases, internal wikis, contracts, technical documents. Where the source material is trustworthy and the question can be answered from it, this approach performs well and is auditable in a useful way: the retrieved documents are visible, so outputs can be checked against their sources.

What it does badly. Anything that requires reasoning across structured data, performing calculations, or applying decision logic. This is a retrieval-and-generation system. It is not a database, a rules engine, or a calculation engine. If the question is "what is our pricing for customer X under contract Y," the system can find the relevant documents, but it commonly struggles to apply the rules consistently. It also tends to perform poorly when the corpus is messy: contradictory documents, outdated material, and undocumented context produce outputs that reflect the corpus's flaws.

The commercial implication. This approach is procurement of organisational knowledge retrieval, not organisational reasoning. The cost structure includes the language model, the retrieval infrastructure, the indexing pipeline, the source corpus management, and the ongoing curation work to keep the corpus trustworthy. Vendors quote licence prices that often exclude the corpus management work, which is an ongoing operational cost. Useful questions to raise cover: who keeps the corpus current, what is the cost of that work, and what happens when the corpus drifts.

The failure mode to watch for. This approach is sometimes presented as if it solves all "use your data" requirements. It does not. A useful distinction is between use cases that are retrieval problems (often well-served by this approach) and use cases that are reasoning or calculation problems (often poorly served by retrieval, regardless of how confident the demo looked). The pre-evaluation framework covers the gating questions to raise with vendors in this area.

Source 3: Structured Business Systems

Many enterprise AI deployments retrieve information not from documents but directly from operational systems. Enterprise resource planning platforms, human capital management systems, customer relationship management platforms, and field service or ticketing systems hold structured, transactional data that documents cannot reliably represent: leave balances, inventory levels, contract status, customer account records, pricing schedules, service histories.

What it does well. Accurate, current retrieval of structured operational data. Where the question is "what is the current leave balance for this employee" or "what is the outstanding balance on this account," connecting to the system of record produces an answer that is precise, current, and traceable to source. The answer comes directly from the operational system, not from a document that may be out of date or incomplete.

What it does badly. Complex queries that span multiple systems or require reasoning across relationships that no single system explicitly encodes. A query asking "which customers under enterprise contracts have outstanding service tickets and active renewal discussions" may require joining data across a CRM, a ticketing platform, and a contract management system in ways that none of those systems directly expose. Queries of this kind often surface integration gaps that pre-deployment assessment did not fully map.

The commercial implication. The cost structure here is primarily integration cost rather than retrieval or modelling cost. Connecting enterprise AI to operational systems involves API access, authentication, data mapping, and ongoing maintenance as source systems are updated, patched, or replaced. Vendors often demonstrate this capability using pre-built connectors for major platforms. Connectors vary in depth and currency. Useful questions to raise cover: which systems are supported, how current the connector is, what data the connector actually exposes, and what the integration cost model looks like when source systems change.

The failure mode to watch for. Vendors presenting connector lists as evidence of integration capability without distinguishing between shallow read-only connections and deep, well-maintained integrations. A connector that retrieves a customer name from a CRM is not the same as a connector that retrieves the full contractual and service history. Demonstrations against realistic use cases, rather than feature lists, tend to produce more useful information about integration depth.

Source 4: Explicit Relationship Models and Knowledge Graphs

A knowledge graph models entities and relationships in an organisation's domain explicitly. Customers, products, contracts, policies, processes, and the relationships between them are encoded in a structured form that can be queried and reasoned over. Language models can be combined with knowledge graphs in several ways, including using the model to interpret natural language and the graph to answer the underlying structured question.

What it does well. Domain-specific reasoning where consistency matters. Pricing logic, eligibility determination, regulatory categorisation, relationship-based queries (which contracts cover which customers under which terms), and any workflow that depends on structured relationships rather than document lookup. Outputs are often deterministic against the graph, which makes them auditable and consistent in ways generative outputs are not.

What it does badly. Anything that requires the graph to be modelled, when it has not been. Knowledge graphs are not free. They require domain modelling, ontology design, ongoing maintenance, and integration with operational systems that hold the source data. If the graph is incomplete, queries return incomplete answers, and incomplete answers in high-stakes workflows can be more problematic than no answer.

The commercial implication. This approach is procurement of structured organisational reasoning, with all the modelling work that implies. The cost structure typically includes platform licences, modelling effort (often substantial in the first year), and ongoing maintenance as the domain evolves. A useful line of enquiry for proposals involving this approach covers: who is doing the modelling work, how is it costed, and what happens to the model as the business changes.

The failure mode to watch for. Vendors offering a knowledge graph platform without acknowledging the modelling work the customer is commonly expected to fund. The platform is the easy part. The model of the domain is the work. A demo over a pre-built sample graph is not evidence the approach will perform for the organisation's specific domain.

Where Should the Answer Come From?

The procurement question is not which architecture pattern to select in the abstract. It is which combination of sources the use case actually requires: model knowledge, organisational documents, operational system data, or structured relationship models. Most non-trivial deployments draw on more than one.

A customer service deployment might use model knowledge for general conversation, document retrieval for policy questions, live system integration for account data, and structured reasoning for eligibility and pricing. The architecture question is which combination, in what proportion, and at what cost.

Four questions cut through most of the decision.

Where does the answer actually live? If it lives in documents, document retrieval is involved. If it lives in operational systems, integration is involved. If it requires applying relationships or rules consistently, structured reasoning is involved. If it requires synthesis or generation that does not depend on organisational data, model knowledge may be sufficient. Most enterprise workflows involve more than one of these.

What is the cost of being approximately right? If the cost is low, generative approaches are often acceptable. If the cost is high, structured approaches earn their additional complexity. Calibrating this honestly, rather than optimistically, tends to produce more accurate cost modelling. Customer-facing pricing decisions, regulatory categorisations, contract interpretations, and clinical or legal-adjacent work often fall in the high-cost-of-error category, regardless of how confident the model's outputs sound.

What does the auditability requirement look like? Where outputs are expected to be explained, traced to source, or defended in a regulatory or contractual context, approaches drawing on document retrieval, operational system data, or structured reasoning provide more useful audit trails than pure generation. Asking vendors to demonstrate the audit trail, rather than describe it, tends to produce more useful information. Accepting "the system is auditable" as a claim without seeing what an audit actually looks like tends to leave governance gaps.

What does the integration and maintenance cost look like over time? Costs vary significantly across source types. Model knowledge is included in the licence. Document retrieval adds indexing and corpus maintenance costs. Operational system integration adds connector and maintenance costs that scale with the number of systems and the pace of change in those systems. Structured relationship modelling adds domain modelling effort and ongoing maintenance. A realistic cost model accounts for all the sources in scope.

These questions sit alongside the broader procurement work covered in the enterprise AI vendor evaluation scorecard, which includes architecture fit as one dimension among several.

What This Means for the RFP

Engaging seriously with the source of answers commonly means surfacing it in the RFP. Generic feature questions will not produce architecture clarity. Specific questions will.

Useful RFP questions include: where does the system get organisational context for the proposed use cases, what infrastructure beyond the model is included in the quoted price, what corpus or system integration maintenance is the customer expected to fund, how are outputs traced to their source, what happens when the underlying source material is contradictory or outdated, and how does the architecture change as the deployment scales.

Vendors with strong, specific answers to these questions are providing architectural clarity. Vendors with vague or feature-shaped answers are obscuring the architecture, sometimes deliberately, sometimes because the system is genuinely drawing only on model knowledge dressed in enterprise framing. Either way, the distinction is worth establishing early in the evaluation.

A Note on Vendor Framing

Several patterns of vendor language are worth noting because they recur across the market and they tend to obscure the source of answers rather than clarify it.

"AI-powered" describes a product that uses some form of AI. It says nothing about where answers come from. This descriptor provides limited architectural information. A more productive enquiry covers what the system actually draws on to answer questions.

"Trained on your data" can mean several things, from genuine fine-tuning of a model on organisational data, to retrieval over an indexed document corpus, to nothing more than the user pasting data into a prompt. The phrase is not standardised across vendors. A useful question covers which one is being offered, and whether the customer's data leaves the customer's environment.

"Reasoning" and "agentic" are often applied to systems that do not reason in any structured sense. They generate text that describes a reasoning process. This is sometimes useful and sometimes not. A useful line of enquiry covers what guarantees the system provides about consistency of outputs across similar inputs.

The pattern across all four: asking vendors to describe where answers come from in operational terms, rather than marketing terms, tends to produce more useful information than accepting product descriptions at face value.

"Integrates with your systems" or "connects to your applications" is a phrase that covers a wide range of actual integration depth. A vendor who claims integration with a major CRM or ERP platform may be reading a small subset of fields via a read-only API, or may have a deep, well-maintained connector that surfaces complex relational data. The claim looks the same from the outside. What varies materially is which objects are accessible, how current the data is, whether writes are supported, and what happens to the integration when the source system is updated. The connector list in a vendor demo is a starting point for the question, not an answer to it.

Why the Decision Cannot Wait

These decisions are easy to defer. They feel technical. They feel like something IT can sort out during implementation. In most cases, they cannot. The architecture an organisation commits to during procurement constrains the use cases it can support, the governance regimes it can apply, the cost curves it will live with, and the exit options it will have. Each of these is a procurement concern, not an engineering one.

The cost of asking these questions early is a slower RFP. The cost of not asking them is a deployment that the organisation either cannot govern, cannot scale, or cannot leave. The trade-off commonly favours asking early.

This article provides general commercial and procurement commentary only and does not constitute legal, financial, or professional advice.