Enterprise AI Governance Considerations for Procurement Teams

Enterprise AI procurement in Australia rarely happens in a governance vacuum. Vendor evaluations, contract negotiations, and deployment decisions commonly sit within a regulatory and governance context that can shape what is procured, how it is used, and what obligations the organisation carries once the contract is signed.

Considering relevant governance requirements early can help procurement teams develop clearer evaluation criteria, identify vendor limitations and reduce issues arising later during legal, privacy, security or risk review.

The governance context also affects what procurement teams are assessing in vendors. A vendor's data handling practices, model update policies, subprocessor chains, and auditability commitments are governance questions before they are contract questions. Procurement teams that understand what a functional AI governance framework looks like are better positioned to evaluate whether a vendor's practices align with it.

This article covers the Australian governance reference points that shape enterprise AI procurement, what a functional governance framework may contain, and the domains that procurement decisions most directly affect.

The Australian Regulatory Context Procurement Teams Are Working Within

Australian enterprise AI procurement operates within a combination of existing Australian laws, sector-specific obligations and voluntary responsible-AI guidance. The most relevant requirements depend on the organisation, its data, its use case, and the people potentially affected. These are not abstract compliance considerations. They can shape what governance requirements procurement teams set in RFPs, what vendor capability assessments focus on, and what gaps may surface during legal review if procurement engages with them later rather than earlier. Two reference points commonly arise in procurement discussions: the Australian Privacy Principles and the Government's Guidance for AI Adoption. Depending on the sector and use case, other regimes, including Australian Consumer Law, work health and safety law, anti-discrimination law, and directors' duties, may also be directly relevant.

The Australian Privacy Principles

The Australian Privacy Principles, contained in the Privacy Act 1988, form an important regulatory baseline for APP entities that handle personal information. They are technology-neutral rather than AI-specific, and several may be relevant where enterprise AI systems handle personal information.

APP 1 requires APP entities to manage personal information openly and transparently, including through a clearly expressed and up-to-date privacy policy. Depending on how an AI system collects, uses and discloses personal information, relevant information-handling practices may also be a factor in privacy documentation and collection notices.

From 10 December 2026, additional APP Privacy Policy requirements will apply where an APP entity has arranged for a computer program to make, or do something substantially and directly related to making, a decision; personal information about an individual is used in operating the program; and the decision could reasonably be expected to significantly affect the individual's rights or interests. Organisations procuring relevant systems may wish to consider whether these requirements apply and what information may be needed from the supplier.

APP 8 addresses the cross-border disclosure of personal information to overseas recipients. Offshore processing, storage or model inference may engage APP 8 where personal information is disclosed to an overseas recipient. Whether a disclosure occurs depends on the particular arrangement, including whether the Australian entity retains effective control of the information.

APP 11 requires organisations covered by the Privacy Act that hold personal information to take reasonable steps to protect it from misuse, interference and loss, as well as unauthorised access, modification or disclosure. Where enterprise AI systems process personal information through prompts, documents or automated workflows, relevant measures may include access controls, secure information-handling procedures, vendor controls, monitoring, incident response, and appropriate retention, destruction or de-identification arrangements. The measures required will depend on the organisation's circumstances and the nature and sensitivity of the information involved.

The APPs do not prescribe specific technical controls. They set principles that a governance framework can translate into operational requirements. Many organisations seek to map governance controls back to relevant APP obligations to support traceability and governance maturity.

The Office of the Australian Information Commissioner has also published guidance specifically on the use of commercially available AI products, recommending product due diligence, privacy impact assessments, assessment of the AI system's access to personal information, human oversight, ongoing lifecycle monitoring, and updates to privacy policies and collection notices. For a procurement-focused governance framework, this guidance is often a more directly applicable privacy reference point than historical ethics principles.

Guidance for AI Adoption

The Guidance for AI Adoption, released by the National AI Centre in October 2025, is now the Australian Government's current guidance for responsible AI adoption. It evolves and simplifies the Voluntary AI Safety Standard (2024), replacing the ten guardrails used in the earlier Standard with six essential practices. It is published as foundations guidance for organisations beginning their AI adoption and implementation guidance, published in May 2026, for governance professionals and technical teams.

The six essential practices are: deciding who is accountable, understanding impacts and planning accordingly, measuring and managing risks, sharing essential information, testing and monitoring, and maintaining human control.

For procurement teams, the Guidance can provide a useful reference point when developing vendor evaluation questions, particularly around supplier responsibilities, transparency, testing, monitoring and supply-chain controls. Procurement teams may ask whether the vendor clearly explains its responsibilities, provides sufficient information about the system's operation and limitations, and supports the organisation's own oversight and accountability requirements. The adopting organisation generally remains accountable for how and where the AI is used; the Guidance is a tool for clarifying how responsibilities are shared across the organisation, its vendors, developers and integrators, not for shifting accountability to the vendor.

The Guidance remains non-binding for private sector organisations, but it is the Australian Government's current guidance for responsible AI adoption, evolving and simplifying both the Voluntary AI Safety Standard and Australia's AI Ethics Principles. Australia's AI Ethics Principles are a set of eight voluntary principles for governments and businesses first published in 2019: human, social and environmental wellbeing; human-centred values; fairness; privacy protection and security; reliability and safety; transparency and explainability; contestability; and accountability. These principles can remain a useful high-level vocabulary for governance discussions, although organisations developing a current framework may find the Guidance for AI Adoption a useful current Australian Government reference point.

Governance Readiness from a Procurement Perspective

When procurement teams assess internal governance readiness before a deployment, or evaluate vendor governance capability during an RFP process, one practical challenge can be knowing what to assess. A policy document alone may not show whether governance arrangements operate effectively in practice. From a procurement perspective, six useful components to consider are scope, accountability, risk classification, operational controls, review processes, and incident and escalation arrangements. These components are not exhaustive, but they provide a practical structure for procurement and vendor assessment.

A policy states what is permitted and prohibited. A framework provides the structure within which policies, controls, accountabilities, and monitoring processes are organised and maintained. An organisation can have a well-written AI policy and no functional governance if those policies are not connected to ownership structures or to any mechanism for detecting when they are not being followed. Procurement teams assessing governance maturity should consider both the policy and whether the supporting governance arrangements operate effectively in practice.

The six components are:

Scope defines which AI systems, use cases, and business processes the framework applies to. Scope defined too narrowly can leave AI activity ungoverned. Scope defined too broadly can create obligations that are difficult to meet in practice. Defining scope involves decisions about what counts as AI for governance purposes, including whether AI features embedded in third-party enterprise software fall within the framework or only standalone AI platforms.

Accountability structure identifies who is responsible for AI governance at each level: the executive sponsor, the operational owner, team-level leads, and individuals with decision authority over AI use. Accountability may be less clear where responsibility is assigned broadly to a function rather than to defined roles or positions. Governance frameworks that specify positions, not just teams, can produce clearer lines of responsibility when something warrants investigation.

Risk classification establishes how AI use cases are categorised by risk level and what governance requirements apply at each level. Lower-risk use cases may involve documentation and basic usage guidelines. Higher-risk use cases may involve formal assessment, legal review, and defined supervisory controls before deployment proceeds. Risk classification is a mechanism that can help make governance proportionate rather than uniform, reducing the likelihood that the framework becomes an administrative burden on low-stakes applications.

Operational controls translate the framework's principles into specific requirements: what data may be submitted to AI systems, how access is managed, what human review takes place before outputs are acted on, how model updates are monitored. This is where applicable legal obligations, government guidance and organisational policies may be reflected in practical controls.

Review and update process establishes how the framework stays current. AI vendor terms change. Regulatory requirements develop. Organisational AI use typically expands after initial deployment in ways that trigger additional obligations. A governance framework without a defined review process can become stale without the organisation noticing.

Incident and escalation process defines what constitutes an AI governance incident, who is notified, how it is investigated, and what authority exists to respond. Governance structures that lack escalation processes can produce slow, inconsistent responses that allow problems to compound before they are resolved.

Information Classification and Personal Information Controls

Procurement decisions about which systems AI integrates with, what data it accesses, and what inputs users can submit are governance decisions as much as they are technical ones. The data classification questions that procurement defines during vendor scoping can influence what controls the governance framework is expected to establish at the AI interface.

AI systems may create different information-handling risks from conventional applications because users can submit free-text prompts and documents, and some tools can access data across connected enterprise systems. These inputs may contain personal, confidential or sensitive information that the user did not recognise or intend to disclose.

This creates a governance challenge that existing information-classification frameworks may need to be extended to address. A governance framework that defines data governance in broad terms without specifying what data types may be submitted to AI systems, through what channels, and under what controls, leaves a practical gap at the point where exposure is most likely to occur.

APP 11 may be relevant where an organisation covered by the Privacy Act holds personal information processed through AI prompts, document processing or related workflows. Governance controls around what information may be submitted, how access is managed, and how prompts and outputs are logged and retained may form part of the technical and organisational measures used to protect that information. The appropriate controls will depend on the organisation's circumstances and the nature and sensitivity of the information involved.

Organisations may address information classification at the AI interface level by defining which data classifications may be processed by which AI systems, under what conditions, and with what controls in place. Organisations operating in sectors where sensitive information is common (health, financial services, legal, government) may choose to treat this as a distinct governance domain rather than a subset of general data governance.

Shadow AI and Unsanctioned Use

Procurement for enterprise AI frequently happens in an environment where employees are already using consumer AI tools for work. This can affect procurement scope, change management planning, and the business case assumptions about adoption. It can also affect governance: a framework designed for a formally deployed AI system does not automatically govern the shadow AI activity that exists alongside it.

Consumer AI tools are widely accessible. Employees at many Australian organisations may already be using them for work tasks, often without organisational visibility or approval. This is commonly referred to as shadow AI: AI use that sits outside the organisation's governance framework because it was never brought within scope.

The governance exposure created by shadow AI is distinct from the risks associated with formally deployed enterprise AI. Data submitted to unsanctioned consumer AI tools may fall outside the organisation's approved data-handling controls. Confidential information, client data, and personal information (referred to in some international frameworks as personally identifiable information or PII, though the Privacy Act uses the term personal information) submitted through consumer interfaces may be processed under terms the organisation has not reviewed and cannot enforce. The organisation's governance framework, however well-constructed, does not govern what it cannot see. The OAIC has specifically recommended that personal information, particularly sensitive information, not be entered into publicly available generative AI tools because of the privacy risks involved.

Detection is difficult. Policy alone may be insufficient without supporting controls, staff awareness and suitable approved alternatives. Organisations may address shadow AI through a combination of measures, including acceptable-use policies, staff education, technical access controls, monitoring, data-loss prevention controls and approved enterprise alternatives. The appropriate combination will depend on the organisation's environment, risk profile and workforce needs. Frameworks that do not address shadow AI govern only a portion of the actual AI activity taking place in the organisation.

AI Inventory and Register

A governance framework applies to the AI systems within its scope. If those systems are not documented, the framework applies to an incomplete and often inaccurate picture of the organisation's AI estate.

An AI inventory or register is a key mechanism through which scope becomes operational. It documents which AI systems are in use, what they are used for, who is accountable for each, and at what risk classification they sit. Without it, risk classification applies in principle but not in practice. Review cycles may not function effectively without a list of systems to review. Incident response can be slower when ownership is unclear.

An AI register exercise can surface more AI use than the organisation initially expected. AI capabilities embedded in enterprise software (productivity suites, CRM platforms, HR systems) may not always be recognised or recorded as AI by the teams using them, and can fall outside the governance framework as a result.

The AI register also serves an acquisition governance function. When new AI systems are evaluated and deployed, the register provides the baseline against which new deployments are assessed, classified, and brought into the governance structure. Frameworks that lack this mechanism tend to treat each new deployment in isolation rather than as an addition to a managed estate.

Supplier Governance

Many organisations obtain significant elements of their AI capability through external vendors. The organisation does not build the model, control the training data, or manage the infrastructure through which inference occurs. This concentration of AI capability in vendor relationships makes supplier governance a core component of a functional AI governance framework.

Supplier governance in an AI context extends beyond standard vendor management. The governance questions specific to AI suppliers include: what rights the vendor retains over data submitted through the platform, whether data is used to train or fine-tune models, how model updates are managed and communicated, which subprocessors handle data and where, and what audit rights the organisation holds.

These arrangements may raise privacy considerations, including under APP 8 where personal information is disclosed to an overseas recipient. Offshore hosting, processing or model inference does not by itself determine whether APP 8 applies. The relevant data flows, recipients, access arrangements and degree of control retained by the Australian entity may need to be assessed in the circumstances.

Supplier governance also covers the ongoing dimension of the vendor relationship post-deployment. Because vendor capabilities, terms and subprocessor arrangements can change over time, supplier governance may need to continue after contract signing. Governance frameworks that include a defined supplier review cycle, with specified triggers and accountabilities, are better positioned to detect and respond to these changes before they create compliance or operational exposure.

Connecting Governance and Procurement

Governance arrangements may be less effective where they are not connected to procurement and acquisition processes. New AI systems could otherwise be introduced without consistent ownership, risk assessment, registration or approval. Linking procurement activities with the organisation's AI register, internal review processes and relevant specialist teams can help bring new deployments within scope before commitments are made.

The connection between governance framework and procurement is addressed in the enterprise AI governance guide: acquisition governance is one of the mechanisms through which the framework can extend to new AI deployments before they are embedded.

What Procurement Teams Can Ask to Test Governance Maturity

The practical test of governance maturity is not a document review alone. It is whether the organisation, or the vendor, can answer a set of operational questions with specificity. Procurement teams that ask these questions during evaluation may surface governance gaps before they become contract problems.

Questions that may help assess whether governance arrangements operate in practice include: which AI systems are currently in scope, who is accountable for each, what controls apply and how they are monitored, when those controls were last assessed, and what happened the last time the governance framework was tested by an actual incident or escalation. The ability to answer these questions with supporting evidence may provide an indication that governance processes are operating in practice rather than existing only in policy documentation.

The connection to procurement can be direct. Embedding governance requirements in vendor assessments, contract terms and deployment conditions can help translate them into practical obligations and controls. Governance requirements that are not embedded in procurement can become retrofit problems. A common risk is that governance gaps identifiable during procurement only become visible after deployment, because procurement was not structured to surface them.

Procurement teams that understand the Australian governance reference points, know what a functional framework contains, and ask the right questions during vendor evaluation are better positioned to avoid this outcome than those treating governance as a post-contract concern.

Important notice: This article is current as at July 2026 and provides general information and procurement commentary only. It does not constitute legal, privacy, regulatory, financial or other professional advice. Laws, regulatory guidance and government policy may change, and their application will depend on the organisation, sector, use case and circumstances. Organisations should obtain advice from appropriately qualified legal, privacy, risk or regulatory advisers where required before relying on this content or making decisions.