The AI Vendor Risk Checklist Nobody Runs Before Signing the Contract

Polished sales presentations and smooth product demonstrations are designed to make AI look magical and effortless. But beneath the gorgeous user interfaces and promises of infinite efficiency lies a complicated and highly volatile technical infrastructure that can expose your enterprise to unexpected risks.

When a contract is signed without thoroughly auditing the vendor’s data pipeline and model lifecycle, organisations effectively hand over control of their operational logic. This is not ordinary software that functions predictably inside a container. It learns, drifts, hallucinates, and potentially leaks the intellectual property that defines your competitive advantage.

“The danger is not only that a chatbot hallucinates. It is that your team cannot say where its prompts went, what the model is, who trained it, which other companies touch the data, whether last month’s version is the one running today, or how to leave without losing a year of work.”

• 01 / Procurement Principles

The first rule: do not buy a black box with a login

Enterprise software procurement has spent thirty years perfecting a standard review model centred on security, availability, and financial stability. However, AI systems break these paradigms. Before analysing a vendor’s tool, you must establish exact clarity on what the system will influence, deciding on its structural operational boundaries.

Buying a simple writing assistant to draft internal emails carries a very different risk profile compared to introducing a resume-screening machine learning model or a customer-support agent with direct refund authority. Matching the procurement rigor to the task’s systemic risk is the foundation of strategic governance.

• 02 / Information Security

1. Data handling: follow the prompt

When an enterprise AI vendor promises that they “do not train our models on your data,” they are answering a single question while ignoring ten others. Data inside modern LLM platforms moves through multiple distinct stages, each requiring strict boundary definitions in your commercial agreements.

1. Absolute Zero-Training Clause: A legally binding prohibition on using any customer-specific data, embeddings, or prompts to train public, base, or adjacent models.

2. Tiered Data Retention: Immediate deletion of prompt caches once a session finishes, with a maximum 30-day window for diagnostic logs.

3. Sovereign Geographic Residency: Explicit restriction of data and model hosting within your specified legislative region.

4. Human Access Restraints: Requirement that vendor technical staff require express, logged permission to access any customer logs.

5. Sub-processor Disclosures: Mandate to notify in writing 30 days before changing underlying LLM APIs or infrastructure hosts.

6. Proactive Incident Disclosures: Fast-tracked 24-hour notification window for any suspected data leak, safety breach, or unauthorized prompt extraction.

• 03 / Technical Lifecycle

2. Model provenance: ask what is actually doing the work

The largest blind spot in AI procurement is ignoring the model supply chain. Many vendors wrap third-party API keys in a gorgeous interface, meaning you are completely dependent on downstream providers for accuracy, availability, and versioning.

10-Item Model Audit Checklist

• 04 / Performance Validation

3. Do not confuse benchmark scores with suitability

A vendor claiming their system scores 90% on standard industry benchmarks is telling you nothing about how it handles your proprietary data format, specialised vocabulary, or unique compliance requirements.

Build a small, proprietary test set

• Collect 30–50 realistic prompt scenarios representative of daily workload.
• Document the ideal, baseline, and dangerous outputs for each scenario.
• Run this set against the vendor’s system during the proof-of-concept phase.
• Rerun the exact same test set monthly to check for model drift or degradation.
• Audit model changes against safety guidelines before live deployment.
• Empower team managers to add new failure cases to the validation set.

• 05 / Strategic Lock-In

4. Exit clauses: your future self is also a stakeholder

AI systems construct invisible, highly effective locks. The moment your employees customise thousands of system instructions, fine-tune internal prompt libraries, and link automated agents together, switching costs skyrocket.

11 Sources of AI System Lock-In

• Custom prompt libraries built by staff.
• Base system instructions and guardrails.
• Fine-tuning training datasets and associated weights.
• User annotation histories and labeling logs.
• Downstream application API integrations.
• Historical audit trails and performance compliance logs.
• Custom safety filters and moderation datasets.
• Vector search indexes and retrieval structures.
• Agentic state histories and memory graphs.
• Custom template configurations and workflows.
• User training hours and specialised system skills.

• 06 / Governance Framework

5. Turn the checklist into a decision

A checklist is useless if it sits in an email. It must be integrated directly into your procurement decision engine, aligning key stakeholders before commercial commitments harden.

• Strategic Action

The question to ask in the room

“If this vendor changed the model, lost access to an upstream provider, suffered a breach, raised prices sharply or became unsuitable for our organisation next year, what would we actually do?”

Answering this single question with concrete, documented migration steps removes ambiguity from procurement. It shifts the discussion from magical thinking to realistic operational risk management. That is what a proper governance framework buys you: not absolute safety, but the strategic control to direct your own technical future.