Showing Posts From

Ai vendor risk

European Company Data in US AI Models: What Your Executives Don't Know Is Already Out

European Company Data in US AI Models: What Your Executives Don't Know Is Already Out

Six months after a European CTO signed off on an AI programme, the DPO sent a single-line email: which data has been sent to the model, where is it stored, and how long is it kept there? Nobody had an answer. The programme had been running on a standard enterprise API agreement. The vendor was reputable. The legal team had reviewed the terms. But the question of what data was leaving the organisation — in what form, to which infrastructure, under what retention policy — had never been mapped. I've seen versions of this conversation more times than I'd like. The AI programme gets approved at the business case level. The vendor agreement goes through procurement. The DPO gets looped in late, or not at all. By the time anyone asks the data flow question seriously, the model has been running for months and the answer is complicated. This isn't a scare piece. It's a map of what actually leaves a European organisation when it runs AI workloads on external models — because the gap between what executives believe is happening and what the contract actually governs is wider than most teams realise. GDPR has been in force since 2018. The enforcement precedent in the AI space is now live. And the organisations that haven't done the data flow mapping are running programmes that haven't been assessed for compliance. What enters a prompt has left the organisation Every query sent to an AI model — every instruction, every document, every piece of text — is transmitted to and processed on the vendor's infrastructure. This is obvious in principle and consistently underestimated in practice. The problem isn't that employees use AI tools. The problem is that the data flowing through those tools hasn't been classified before it flows. When a salesperson pastes a customer contract into a prompt to generate a summary, that contract has left the organisation. When an HR manager uploads a performance review to ask for a draft response, that review has left the organisation. When a finance analyst feeds a quarterly forecast into a model to restructure the narrative, that forecast has left the organisation. None of this requires a policy violation. It happens in ordinary, productive use — which is exactly why it's difficult to control retrospectively. Most enterprise AI APIs retain prompt data for abuse monitoring purposes unless the customer has explicitly opted out — and opted out in writing, in a way the contract records. Retention periods vary by vendor. They are subject to change. I have reviewed enterprise AI agreements where the data retention section was three lines long and referred to a separate policy document that had been updated four times in eighteen months. The business owner who approved the tool had never read either version. Data classification before any AI tool goes into production is not optional — it is the prerequisite. Without knowing what data is sensitive, personal, confidential, and commercially restricted, there is no basis for deciding what can enter a prompt. Most European enterprises have data classification policies. Most of those policies were written before AI prompt data was a category that needed to be governed. Personal data in prompts — minimisation is not enough When the data entering a prompt includes names, email addresses, identification numbers, health records, or financial information, the transfer isn't just a data governance concern — it is a GDPR event. Personal data has moved to an external processor, potentially outside the EU, under whatever transfer mechanism (or lack of one) applies to that vendor relationship. The standard response inside most European enterprises is to invoke data minimisation: GDPR's Article 5(1)(c) principle that personal data should be adequate, relevant, and limited to what is necessary. In theory, this means sending only the fields required for the AI task. In practice, it almost never works that way. Teams send full customer records because the model produces better outputs with more context. HR workflows send complete personnel files because partial information leads to incomplete responses. Customer service AI is fed entire interaction histories because isolated records produce generic answers. The business logic for sending more is always compelling, and the minimisation principle gets applied at the policy level while the actual data flows ignore it. Even when minimisation is applied properly, it is not a compliance answer for personal data. A dataset that contains only a customer's name, email address, and account status is minimal — and it is still fully personal data under GDPR. Minimising what you send does not change the legal classification of what arrives at the model. The transfer still requires a lawful basis. The vendor still needs to be a compliant data processor. The cross-border transfer rules still apply. The only way personal data falls outside GDPR scope when it reaches an external AI model is if it has been genuinely anonymised — not minimised, not pseudonymised, but anonymised to the standard where re-identification is not reasonably possible for anyone, including the model provider. That standard is far harder to meet than most teams assume, and AI models introduce a specific re-identification risk through pattern learning that conventional processing does not. This is covered in full in a separate post in this series. The practical position for European enterprises: names, email addresses, and any other direct identifiers should not enter an external AI model unless a valid GDPR lawful basis, a compliant data processing agreement, and a valid cross-border transfer mechanism are all in place — and verified, not assumed. The metadata picture your vendor is accumulating Prompt content is the obvious exposure. Usage metadata is the one that organisations consistently miss. AI vendors collect usage patterns: query frequency, document types, feature usage, workflow structures, which teams use the model for what. Individually, these data points look low-sensitivity. Collectively, over time, they build a detailed picture of how the organisation operates — what it's working on, where it's directing AI resources, which functions are scaling their usage and in what direction. This is a competitive intelligence risk that exists independently of whether any personal data is involved. A vendor who can see that a company's legal team started using the model heavily in March, its M&A function spiked usage in April, and its HR team has been running high volumes of termination-related drafts since May has learned something meaningful — without any named individual appearing anywhere in the data. Enterprise agreements typically don't restrict vendors from using aggregated, anonymised usage data for product improvement and benchmarking. The data processing agreement and the main commercial contract often address prompt data and metadata under different provisions, negotiated at different points in the contracting process, with different protections attached to each. The metadata protections are almost always weaker. Read both documents carefully — not just the DPA summary the vendor sends as a compliance checkbox. The training data assumption most enterprise teams get wrong There is a persistent assumption in European enterprises that the enterprise version of an AI tool doesn't use customer data for training. Sometimes this is correct. Often it is an assumption that hasn't been verified against the current version of the data processing agreement. Consumer-grade AI tools — the non-enterprise versions of products your employees are also using — may by default use user inputs to improve the underlying model. This is processing for a new purpose. Under GDPR, processing for a new purpose requires a separate lawful basis. When an employee uses the consumer version of a tool on work data because it's faster or the enterprise licence hasn't been set up on their machine, the organisation has a problem it didn't create deliberately but owns entirely. Enterprise agreements typically include data processing agreements that prohibit training on customer data. Three things determine whether that prohibition is actually protecting the organisation: whether it applies to all data types or only certain categories; whether it covers all infrastructure the vendor uses, including subprocessors; and whether the DPA reflects the current model architecture. I have seen enterprise agreements where the DPA predated a major vendor platform change by over a year. The prohibition on training was still in the original terms — which no longer accurately described what was happening with the data. The check is straightforward. Request the current, signed DPA. Confirm it covers training, all subprocessors, and the current model version. Don't assume continuity because the vendor's name hasn't changed. When model outputs carry a breach risk This risk sits with the deployer — and it is the one most European organisations haven't included in their AI risk registers. If a model has been trained on data that included personal information, it can under certain conditions reproduce elements of that information in outputs. Not reliably, not on demand — but under specific prompting patterns, models have been shown to surface fragments of training data. If European employee or customer data was part of the training set, and the model surfaces it in a response, the deploying organisation has a data breach on its hands. Under GDPR, that breach sits with the deployer. The model provider may carry liability, but the deployer cannot transfer its own. For foundation models from major providers, the deployer typically has no visibility into training data composition. The risk is difficult to quantify and impossible to eliminate entirely. For systems where the deployer has fine-tuned or augmented a foundation model with internal data — retrieval-augmented generation, fine-tuning on internal documents, embedding proprietary content — the deployer has direct control over what went in. That control increases responsibility. Before augmenting or fine-tuning a model with internal data, the architecture review needs to answer: what personal data is in these documents, what happens if the model later surfaces fragments of it, and what technical controls reduce that risk? This is a question most teams are not asking at the design stage. It becomes the question they wish they had asked after an incident. What the enterprise agreement covers — and what it doesn't The enterprise agreement creates contractual obligations. It doesn't create infrastructure controls. The data processing agreement requires the vendor to handle data in certain ways. It doesn't prevent US law enforcement from issuing a lawful demand to a US-incorporated vendor under US law — a structural problem I address in the next article in this series, focused specifically on cross-border transfer risk. What the enterprise agreement also doesn't replace is the audit trail that European regulators now expect to exist. The June 2025 guidance from Germany's data protection conference (DSK) requires documentation across the AI system lifecycle. That documentation needs to exist before regulators ask for it — not in the forty-eight hours after they do. For European companies with German operations, this requirement is not optional and it is not the vendor's responsibility to create. The enforcement precedent is live. A major AI provider was fined €15 million by Italian regulators in December 2024 for GDPR violations — failure to establish a lawful processing basis and inadequate transparency around how user data was handled. European regulators now have precedent, process, and political pressure to continue in this space. The organisations currently running AI programmes without a data flow map are running programmes that have not been assessed for compliance. The gap between those two states is not a legal technicality — it is the gap between a defensible position and one that cannot be defended when examined. What to take from thisMap data flows before deployment, not after. Every AI tool processing external data should have a documented flow — what enters, what is retained, where it's stored, for how long — completed before the tool goes live, not reconstructed months later. Read the current DPA, not the version in the procurement file. Data processing agreements are updated. Request the current signed version and confirm it covers training, subprocessors, and the current model architecture. Classify data before it enters a prompt. Sensitive, personal, commercially restricted, and confidential data should be classified before it enters any AI tool — not assessed after six months of use. Do not treat data minimisation as a substitute for anonymisation. Sending fewer fields does not change the legal classification of what arrives at the model. Personal data that has been minimised is still personal data — names, email addresses, and identifiers stripped down to the "minimum necessary" still carry full GDPR obligations. If personal data cannot be genuinely anonymised before it reaches an external model, the transfer needs a lawful basis, a compliant DPA, and a valid cross-border mechanism — not a smaller dataset. Treat usage metadata as a competitive risk, not just a data protection issue. Understand what your vendor collects on usage patterns, what restrictions apply, and what they're permitted to do with aggregated data. Enforce a clear policy on consumer versus enterprise tool versions. If employees have access to both, the policy needs to specify which applies to which data — because the data handling terms are materially different. Include model output re-identification in your AI architecture review. For any system fine-tuned or augmented with internal data, answer the re-identification risk question before build — not after the system is in production.GDPR has been in force since 2018. The data protection gap in AI programmes isn't a new legal problem — it is an existing problem that the pace of AI adoption is making visible at scale. Most of what European companies need to do to close this gap doesn't require waiting for new regulation. It requires applying existing obligations to a class of tools that moved faster than the governance frameworks around them.

Read full article