The AI Tools Your Employees Are Using With Your Data

The AI Tools Your Employees Are Using With Your Data

The standard framing for AI governance starts with the question of which tools to approve. That is the wrong starting point. The better question is: which tools are already in use, with what data, under what terms? By the time most organizations start building an AI governance framework, their employees have already made a fairly coherent set of tool choices. They have picked the tools that solve their immediate problems. They have not, in most cases, read the privacy policies or data processing terms. And because nobody told them not to, they have been using company data freely. A CIO who wants to get ahead of this — or who wants to manage it after the fact — needs a clear picture of the tool landscape and an honest assessment of where the data risk actually concentrates. Not all unsanctioned AI tools carry the same risk. Understanding the difference is where governance work should start. The tool categories and what they mean for data General-purpose AI assistants This is the highest-volume category. Consumer versions of large language model interfaces are used daily by employees across functions — drafting communications, summarizing documents, answering domain-specific questions, structuring thinking. The use is frequent, the content fed in is varied, and the data handling terms depend entirely on whether the employee is using a consumer or enterprise account. The specific risk here: consumer-tier accounts with default settings often permit the vendor to use interaction data for product improvement. The same vendor's enterprise tier typically does not. Most organizations have no visibility into whether employees using these tools are on a consumer or enterprise tier, and many are on a consumer tier simply because it was free and faster to start. Productivity AI features in existing software Word processing, spreadsheets, presentation tools, email clients, and project management platforms increasingly include AI features — often activated via a premium license or a setting employees can enable without IT involvement. The risk here is different from standalone AI tools: because these features exist inside software the organization already uses, they often fly under the radar of any AI tool review. The data handling terms for AI features embedded in existing software are usually governed by the same agreement covering the base product, but with additional clauses for the AI component that many organizations have not reviewed since they were added. These clauses deserve explicit attention. Specialist function tools Legal AI tools, sales intelligence platforms, HR tools, finance automation assistants, coding assistants, market research tools — these are purpose-built AI products targeting specific professional functions. They tend to be adopted department by department, often through a free trial that converts to a team subscription without going through central IT. The data risk with specialist tools is often higher than with general-purpose ones, for a specific reason: the content fed into specialist tools tends to be more consistently sensitive. Legal teams feed contracts. Finance teams feed financial models. Sales teams feed client data and deal structures. The tool is designed for that content, which means employees use it confidently and at volume. AI-powered integrations and automation platforms Workflow automation tools, AI connectors between SaaS platforms, and integration layers that use AI for data transformation or decision-making sit in a category that CIOs are least likely to have visibility into. These tools often operate in the background — processing data as part of an automated flow rather than through a direct user interaction — and their data handling terms are buried inside integration documentation that nobody reads. The risk with automation platforms is not necessarily higher than with interactive tools, but the visibility is lower. When a human pastes text into an AI tool, there is at least a moment of conscious choice. When an automated workflow passes data through an AI component as part of processing, there is no such moment. The risk factors that actually matter When assessing the data risk of any specific tool category, there are four factors that determine how much it matters. What data flows through it. The highest risk is where the most sensitive data concentrates: client information, financial projections, legal material, personal data. This varies by tool and by how a specific team uses it. What tier the organization is on. Enterprise agreements typically include data processing terms, exclusions from training use, and deletion rights that consumer tiers do not. A tool is not inherently high-risk or low-risk — the tier and the agreement terms are what determine the actual data handling. Whether a data processing agreement exists. For any tool processing personal data of EU residents, a data processing agreement is a legal requirement under data protection regulation, not a nice-to-have. Many organizations are operating without these agreements in place for tools their employees use every day. How much volume is flowing through it. A low-use tool with poor data terms is a lower priority than a high-use tool with poor data terms. Volume matters. The tools employees reach for first, most often, at highest volume are where the exposure is concentrated. What a CIO needs to do before writing a policy Policies written without a clear picture of the current state tend to be wrong in two ways: too restrictive in areas where the risk is manageable, and silent on areas where the risk is real. Getting the picture right first makes the policy more useful. That means running a discovery exercise that goes beyond the IT procurement system. Talk to department heads about what their teams use. Survey employees. Analyze network traffic for connections to known AI tool endpoints. The goal is a realistic list of tools in active use, categorized by function and frequency. For each tool, determine what tier the organization is on — enterprise or consumer — and whether a data processing agreement exists. This is the most important variable in understanding the actual data handling exposure. From there, prioritize remediation by volume and sensitivity. The tools that process the highest volume of the most sensitive data under the least favorable terms are the first order of business. That might mean migrating employees from a consumer tier to an enterprise tier of the same tool. It might mean negotiating a data processing agreement with a vendor. It might mean replacing a tool with an approved alternative. The classification that comes out of this exercise — which tools are approved at which tier for which data types — is what the policy should be based on. Policies that precede this exercise tend to produce compliance theater rather than actual risk reduction. The conversation with department heads This is where the process usually gets uncomfortable. When a CIO discovers that a department has been using an unsanctioned AI tool with client data for the past year, the instinct is often to shut it down immediately. That is rarely the right response. Abrupt prohibition creates resistance and drives use underground. It also signals that the governance process is about compliance rather than risk management, which damages the working relationship the CIO needs to make future governance effective. The better approach: treat the discovery as information rather than a violation. Understand what the tool is being used for, what problem it solves, and what the actual data exposure has been. If the tool can be moved to an enterprise tier with appropriate terms, do that quickly. If it needs to be replaced with an approved alternative, make the transition timeline reasonable and the approved alternative usable. Department heads whose teams are using shadow AI tools are not adversaries. They are telling you, through their behavior, what the organization's official tooling is failing to provide. The policy conversation goes much better when it starts from that acknowledgment. What to take from thisMap what tools are in active use before designing any AI tool governance policy. The gap between what IT has approved and what employees are actually using is almost always larger than expected. For each tool, determine whether it is in use on an enterprise or consumer tier. That distinction drives most of the material data handling difference. Check whether data processing agreements exist for tools processing personal data. This is a current legal obligation, not a future aspiration. Prioritize remediation by volume and sensitivity: high-use tools handling sensitive data under weak terms first. Treat departments using unsanctioned tools as providing product feedback. Understand why they chose the tool before deciding how to respond.The CIOs who manage this well are not the ones with the strictest policies. They are the ones who ran the discovery work, understood what was actually happening, and built governance around the real picture rather than the one they assumed existed.

Read full article
What Data Leaves Your Organization Every Time Someone Uses an AI Tool

What Data Leaves Your Organization Every Time Someone Uses an AI Tool

Most organizations operate under a working assumption that their data is contained. Files live on approved systems. Emails go through monitored infrastructure. Cloud storage is access-controlled. The perimeter is imperfect, but it is at least visible. AI tools have quietly dismantled that assumption. Not through a breach. Through normal, sanctioned-feeling use. Every time an employee types a prompt into a large language model, attaches a document for summarization, or pastes a block of text for analysis, that content leaves the organization's infrastructure and enters a third-party system. The employee does not experience this as data transfer. They experience it as using a tool. But the data has moved, and where it goes, how long it stays, and what is done with it depends entirely on terms most organizations have never reviewed. What "data leaving the building" actually means The framing matters here, so I want to be precise. When I say data leaves the organization, I mean three distinct things. First, the input reaches the vendor's infrastructure. The prompt, the document, the pasted text — all of it travels to servers the organization does not control, under security and access policies the organization did not set, in jurisdictions the organization may not have mapped. Second, the vendor processes and stores that input for some period. The length and purpose of storage varies dramatically by product and by the specific agreement in place. Some vendors retain inputs for a defined period for abuse prevention. Some retain them longer for product improvement. Some will, under certain terms, use them to improve future model versions. The defaults on this vary and are not always what organizations assume. Third, the output the model generates may itself be derived from patterns the model learns over time. This is the mechanism that tends to unsettle executives most when they understand it, though the practical risk here is more nuanced than the headline version usually suggests. The part that matters most in practice is the first two: the content reaches third-party infrastructure, and its fate is governed by the vendor's policies, not yours. The content that tends to flow through AI tools This is worth spending time on, because organizations that have audited actual AI tool usage consistently find that the content flowing through consumer and productivity AI tools is more sensitive than they assumed. Strategy and planning documents. Employees use AI tools to refine presentations, summarize options, and draft documents for leadership review. The source material they feed in frequently includes internal plans, financial projections, and competitive analysis. Client and customer information. Sales teams use AI assistants to draft proposals and account summaries. Support teams use them to summarize case histories. Analysts use them to structure reports. Client data is routinely included, often without a deliberate decision to include it. Legal and contractual material. Lawyers and procurement teams use AI tools to summarize contracts, identify key clauses, and compare terms. Contract text often contains commercially sensitive information that neither party intended to share beyond the two signatories. HR and personnel data. Managers use AI tools to draft performance reviews, restructuring communications, and offer letters. The inputs frequently include specific salary information, performance ratings, and personal circumstances. None of these employees are being careless. They are using AI to do their jobs. The exposure is a product of normal behavior, not negligence. Where the data goes: the three mechanisms Processing for the immediate request. This happens in every interaction, by definition. The data reaches the model, the model generates a response, and the exchange is complete from the user's perspective. What happens after that depends on the vendor. Retention for operational purposes. Most AI services retain some record of interactions for a period — to detect abuse, to provide conversation history to the user, or to meet regulatory requirements in certain jurisdictions. The retention period and what the organization can do about it (deletion requests, data portability) varies significantly and is usually defined in the data processing agreement or privacy policy. Use for model training and improvement. This is the term that gets the most attention, and for good reason. Some AI products, particularly consumer-grade versions of enterprise tools, include default settings that allow the vendor to use interaction data to improve the model. The important nuance: enterprise agreements frequently exclude this, while consumer free tiers often include it. The problem in most organizations is that employees are using a mix of both, and nobody has mapped which is which. The distinction between enterprise and consumer tiers on this specific point is where most of the real exposure sits. An employee using an enterprise-licensed product with a properly negotiated data processing agreement is in a materially different position than an employee using the same vendor's free consumer product with default settings. The output is functionally identical. The data treatment is not. What the CTO and CIO actually need to understand The question is not whether AI tools create data exposure — they do, by design, in the same way any cloud service does. The question is whether the organization's data exposure through AI tools is understood, consented to, and consistent with its regulatory and contractual obligations. That requires knowing three things you probably do not know right now. What tools are actually in use. Not just the ones IT has approved — all of them. This means running discovery before designing governance. Most organizations that do this discovery find a longer list than they expected. What tier of each tool is in use. The enterprise agreement and the free consumer version of the same product often have dramatically different data processing terms. This distinction matters for training data use, retention, and deletion rights. What the data processing terms actually say. Not the marketing language about being "privacy-first" or "enterprise-grade" — the actual data processing agreement. Specifically: what the vendor can do with inputs, how long they retain them, what the organization's rights are around deletion, and where the data is processed. Most organizations have answered none of these questions systematically. The CIO knows what is in the procurement system. The CTO knows what is in production. Neither has a complete picture of what is happening between individual employees and third-party AI services. The regulatory and contractual layer Data flowing to AI tools does not exist in a vacuum. It intersects with existing obligations. If the organization operates under data protection regulation, any transfer of personal data to a third-party processor requires a legal basis and, in many jurisdictions, a data processing agreement that specifies how the processor may use the data. AI tools that process personal data — and most enterprise use cases involve at least some personal data — need to be assessed against these requirements. If the organization has contractual confidentiality obligations to clients, those obligations typically extend to how client data is handled regardless of the tool involved. A consultant uploading client strategy documents to an AI summarization tool without a data processing agreement in place may be in breach of their client agreement, regardless of whether the AI tool's terms are otherwise acceptable. These are not hypothetical risks. They are existing obligations that most organizations have not mapped against their AI tool usage. What to take from thisAudit what AI tools are in active use across the organization before designing any data governance response. The list will be longer than IT's approved toolset. Distinguish between enterprise and consumer tiers. The same tool can have dramatically different data processing implications depending on which version employees are using. Read the data processing agreements — specifically the sections on input retention, training use, and deletion rights. Do not rely on the vendor's marketing language. Map AI tool usage against existing data protection and client confidentiality obligations. The intersection is almost certainly not clean. Build a disclosure and classification step into any AI tool approval process: what categories of data can employees use with this tool, under what conditions?The data exposure from AI tools is not a future problem to prepare for. It is a current condition to understand. The organizations that handle this well are not the ones with the most restrictive policies — they are the ones that ran the discovery work, understood what was actually flowing through which tools, and made deliberate decisions about what that meant for their obligations.

Read full article
The Data Quality Problem Nobody Puts in the Deck

The Data Quality Problem Nobody Puts in the Deck

Early in a data program at a large insurer, I asked the head of the business data office what percentage of their claims data they considered clean. She said, without hesitation, about 80 percent. When we actually looked — ran completeness checks, consistency validations, temporal analysis, cross-referenced against the systems of record — the number was closer to 40 percent. And that was by a generous definition of "clean." The 40-point gap was not the result of negligence. It was the result of the way enterprise data accumulates: through system migrations that didn't fully reconcile historical records, through form field changes that made old values semantically incompatible with new ones, through operational shortcuts that were rational at the time and invisible until someone tried to use the data systematically. This pattern — a significant gap between what the business believes about its data and what the data actually contains — is present in virtually every large enterprise I've worked with. What changes is how much it matters. In most operational contexts, it doesn't matter much. In AI programs, it matters enormously. Why the gap doesn't surface until you try to train Data discovery processes reveal the problems that can be found by looking at data directly: missing values, obvious format inconsistencies, clearly duplicated records. What they don't reveal are the problems that only become visible when you try to do something with the data. Semantic inconsistency is one example. A "claim status" field that takes values of "open," "pending," "in review," and "active" might look fine in discovery. The problem emerges when you try to build a model that predicts claim duration and discover that "pending" meant two different things before and after a system migration five years ago. The model learns from the historical pattern and produces predictions that are systematically off for a segment of claims because the label meant something different during the training window. Temporal invalidity is another. Features constructed from historical data often embed assumptions about time that are violated in ways that aren't visible until you start building features. A "days since last contact" feature that looks like a useful signal turns out to encode data entry behavior rather than customer behavior — the field was populated differently in different branches, and the differences correlate with branch-level outcomes rather than customer-level ones. These problems don't show up in a data profile. They show up in model validation, in production performance anomalies, and in the kinds of questions that domain experts ask when they look at model outputs that seem technically sound but operationally wrong. The four failure patterns Completeness gaps. Missing data is the most visible quality problem and usually the best understood. But completeness is less binary than it appears. A field that's 95% complete might have its 5% missingness concentrated in the segment of the data the model most needs to reason about — a specific customer segment, a specific time period, a specific geography. Aggregate completeness metrics hide distributional missingness that can create systematic model errors invisible until production. Consistency failures. Data that means different things in different records, or that's been encoded differently across systems, is the failure pattern that's hardest to detect and most dangerous for model training. Consistency failures are common at integration points — where data from one system is loaded into another — and at migration boundaries, where historical records under an old schema are mapped to fields in a new schema. The mapping logic that seemed sensible at migration time often introduces subtle distortions that aren't documented and don't announce themselves. Temporal drift. The relationship between data and the world changes over time. Customer behavior changes, market conditions change, business rules change. A model trained on data from three years ago has learned from a world that no longer exists in important ways. This isn't a data quality problem in the traditional sense — the data accurately reflects what was true at the time — but it creates a model that doesn't reflect current reality. Temporal drift is the most common reason AI models underperform in production relative to testing, and it's consistently underweighted in most data quality assessments. Labeling errors. For supervised learning problems, the quality of the labels — the ground truth the model is learning from — determines a ceiling on model quality that no algorithm can overcome. Label quality is frequently taken for granted in enterprise AI programs because the labels come from an existing operational system that the business trusts. But operational labels are a product of the processes that generated them, and those processes have their own inconsistencies. Claims classified as fraudulent by one review team using one set of criteria, then reclassified by another team using updated criteria, produce a label set that encodes inconsistency as signal. The model learns from that inconsistency and reproduces it at scale. How it kills AI ROI The mechanism by which data quality degrades AI ROI isn't usually a catastrophic failure. It's a gradual tax on every part of the program. Model performance caps are lower than they should be, which means the business case is harder to achieve. The team spends more time on data remediation than on model development, which means the delivery timeline extends. Retraining cycles are more expensive because the data pipeline that feeds them is brittle, which means the operational cost is higher than projected. And when the business starts to see outputs that seem wrong — where the model disagrees with what an experienced practitioner would say — trust erodes in ways that are very difficult to reverse. The cumulative effect is hard to quantify precisely, but a program that expected an eighteen-month path to production value and took thirty months instead, with model performance fifteen points below the initial projection, is not unusual when data quality problems were underassessed at the start. The feature engineering temptation The engineering response to data quality problems is usually feature engineering workarounds: bridge tables, deduplication logic, reference data lookups, semantic normalization applied in the feature construction layer. These work. I've used them. But they're a form of debt that compounds. A feature pipeline that applies complex normalization to reconcile inconsistent reference datasets has to be maintained for as long as the model runs. When the underlying data changes — and it will — the workaround may silently break, degrading model performance without triggering an error. And every new model that uses the same data inherits the same problem independently, solving it in its own way, creating a portfolio of different workarounds for the same underlying issue. Feature engineering can bridge a data quality gap temporarily while the underlying issue is being addressed. It's not a substitute for addressing the underlying issue. The difference matters when you're building the fourth model on the same data foundation that the first three models already patched around. What a real data quality assessment looks like The most useful question to ask before starting an AI program is not "do we have the data?" Almost every enterprise has data. The question is "is the data we have sufficient to support the model we need to build?" A data quality assessment designed around that question covers: the completeness and consistency of each field the model will use, the temporal validity of the training window, the label quality for the target variable, and the regulatory and compliance status of the data being used for training. For a focused use case, this takes two to four weeks. It produces a realistic view of what remediation work is needed before model training can begin, what workarounds are viable in the short term, and where the data gaps are severe enough to require a different use case or a longer pre-program phase. That view has a cost — it may reveal that the program timeline needs to move right, or that a use case needs to change. But it's a cost paid once, upfront, with full information. The alternative is paying a larger cost spread across months of rework, performance shortfalls, and stakeholder trust that's harder to rebuild than it was to lose. The data quality problem almost always goes into the deck eventually. The question is whether it goes in at the start, when there's still time to do something about it, or at month fourteen, when everyone is looking for someone to blame.

Read full article
AI Risk Is Not IT Risk: Why Your Existing Frameworks Are the Wrong Starting Point

AI Risk Is Not IT Risk: Why Your Existing Frameworks Are the Wrong Starting Point

When a board's risk committee receives an AI risk update, it typically looks like an IT risk update with a new heading. Threat categories, control owners, residual risk ratings, escalation thresholds. The framework is familiar because it was borrowed from cybersecurity and data privacy, the last two major technology risk domains the business had to absorb. The problem is that AI risk is structurally different from IT risk. Not more complex in a general sense — different in kind. The threat model is different, the failure modes are different, and the controls that work for one don't work for the other. Running AI risk through an IT risk framework is like using a fire suppression system to manage flood risk. Both are disaster scenarios. The equipment doesn't transfer. Most organizations won't discover this until something goes wrong. At that point, the questions — who was monitoring this, what controls were in place, who was accountable — will point back to a governance structure that was never designed for the problem it was supposed to prevent. Why IT risk frameworks break on AI IT risk is fundamentally binary. A system is breached or it isn't. Data is exfiltrated or it isn't. A control is working or it has failed. The threat model assumes a boundary — a perimeter — and the job is to defend it. When the perimeter is breached, you know. There's an incident. Systems go down, alerts fire, logs show the intrusion. AI risk doesn't work that way. The most dangerous AI failure modes are probabilistic and continuous. A model doesn't fail — it drifts. Its performance degrades incrementally as the world changes and the training data becomes less representative. There's no incident, no alert, no obvious moment when things stopped working. The model continues to produce outputs that look correct. Business processes continue to run on those outputs. Trust accumulates in a system that is quietly becoming less reliable. This is the silent failure mode that IT risk frameworks have no vocabulary for. A cyberattack is visible by design — the attacker wants access. A degrading model is invisible by design — there's nothing malicious, just a slow divergence between what the model learned and what the world currently looks like. The second structural difference is the threat surface itself. IT risk focuses on external threat actors: attackers, phishing campaigns, vulnerability exploitation, insider threats. AI risk introduces an entirely different threat surface: data poisoning during training, adversarial inputs designed to manipulate model outputs, emergent behaviors that weren't anticipated in testing, and feedback loops where a model's outputs influence the data it will be trained on next. These are not threats a penetration test catches. They're not threats a SIEM detects. They require a different monitoring approach and a different set of controls. The control gap IT risk controls are designed around access, authentication, encryption, patching, and incident response. These are the right controls for IT risk. Most of them are irrelevant to AI risk or address a small subset of it. The controls that actually matter for AI risk are: Model monitoring. Tracking model performance over time against defined thresholds — not just system availability, but whether the model's outputs remain accurate and calibrated. This requires ground truth data, a measurement cadence, and someone accountable for reviewing the results. Data provenance and lineage. Understanding where training data came from, what transformations it went through, and whether those sources have changed. A model trained on data that has since changed in composition is a model with an unknown risk profile. Input validation. Monitoring what the model is being asked to score in production, and detecting shifts in the input distribution that may indicate the model is being asked to operate outside its training domain. This is not the same as network traffic monitoring. Explainability requirements. For consequential decisions — credit, insurance, hiring, medical — the ability to explain why the model produced a specific output for a specific input is both a regulatory requirement in many jurisdictions and a basic operational requirement for incident response. Decommissioning criteria. A defined threshold at which a model is pulled from production, not left running after it has degraded past the point of reliability. Most IT governance has no equivalent because software doesn't degrade the way models do. None of these appear in a standard IT risk control library. They require a purpose-built AI risk framework or significant extension of existing frameworks — not a mapping exercise that tries to force AI risk into existing categories. The accountability mismatch IT risk has clear ownership because IT risk maps to IT infrastructure. The CISO owns the security controls. The CTO owns the infrastructure. Accountability lines are established. AI risk doesn't map to the same functions. The data science team that built the model may not have the operational accountability for what it does in production. The business unit that requested the model may not have the technical literacy to understand the risk it carries. The IT function that runs the infrastructure may have no visibility into whether the model is performing correctly. The accountability gap in AI risk is not a people problem — it's a design problem. The organization hasn't defined who owns AI risk at the model level, and existing governance structures don't force that definition because they weren't built with AI in mind. Good AI risk governance requires a named risk owner for each production model: someone accountable for monitoring its performance, escalating anomalies, and making the decommissioning call when thresholds are breached. That's a different role from anything in a standard IT risk organization chart. What the board's AI risk conversation needs to cover Most board-level AI risk conversations are about whether the organization has an AI risk policy. That's the wrong question. The right questions are operational: Which AI systems are currently making decisions that affect customers, employees, or revenue — and what is the monitoring status of each? What is the defined performance threshold below which each system would be paused or replaced? When did the board last receive an update on actual model performance, not just program status? What is the process for escalating an AI failure to board level, and has it ever been tested? These are not technology questions. They're governance questions that require technology inputs. The reason they're not being asked is that the board's risk framework doesn't prompt them — because the framework was built for a different threat model. The regulatory layer The EU AI Act introduces a formal risk taxonomy for AI that is worth understanding even for organizations not primarily subject to EU law — because it's the clearest articulation currently available of what AI-specific risk management looks like at a regulatory level. The Act's risk tiers map roughly to the severity and reversibility of consequences: prohibited uses (facial recognition in public spaces, social scoring), high-risk uses (credit, hiring, education, law enforcement support), and lower-risk uses with transparency requirements. High-risk systems require conformity assessments, ongoing monitoring, human oversight provisions, and auditability of training data. Whether or not an organization is directly subject to the Act, those categories are a useful starting framework for internal AI risk classification. The question "would this qualify as high-risk under the EU AI Act?" is a reasonable first filter for deciding which AI systems need the most rigorous governance treatment. Organizations in financial services and healthcare already have sector-specific AI risk guidance from their regulators — the EBA, PRA, FDA, and others have all issued guidance that is more detailed and more prescriptive than general enterprise risk frameworks. These are the relevant starting points for organizations in those sectors, not the enterprise risk framework that was designed before AI was a material concern. Building an AI risk framework from scratch is a significant undertaking. Borrowing one from IT risk is the path of least resistance, and the path most likely to leave the organization exposed when something goes wrong. The frameworks aren't interchangeable, and the gap between them is where most AI governance failures currently live.

Read full article
The AI Audit Your Board Should Be Asking For (But Probably Isn't)

The AI Audit Your Board Should Be Asking For (But Probably Isn't)

When organizations commission audits, they tend to know what they're looking for. A financial audit looks for misstatements. A cybersecurity audit looks for vulnerabilities. Both have established methodologies, credentialed practitioners, and a clear output format. No equivalent exists yet for AI — and the absence is starting to matter. Most boards that have approved AI investments can answer some questions about them: what budget was committed, which vendor was selected, whether the program is on schedule. Very few can answer the questions that actually determine whether the organization's AI exposure is understood and managed: which AI systems are currently making decisions that affect customers, employees, or revenue, who is accountable when those decisions are wrong, and would anyone be able to explain a specific bad outcome if asked to by a regulator or a plaintiff's attorney. The audit that answers those questions isn't a technical audit. It's a strategic one — a systematic review of what the organization is actually doing with AI, whether the governance structures in place are real rather than documented, and whether the accountability chains hold under scrutiny. What a strategic AI audit is A strategic AI audit is a decision audit. It asks: what decisions is AI making in this organization, and is the governance around those decisions adequate? This is different from a technical audit, which asks whether AI systems are built correctly. It's different from a compliance audit, which asks whether documentation requirements have been met. And it's different from a security audit, which asks whether AI infrastructure is protected from external threats. The strategic audit asks the governance question: if something goes wrong with an AI-driven decision, does the organization know what happened, does someone own it, and is the board in a position to account for it? In my experience, the answer to at least one of those three questions is "no" in most organizations that haven't specifically designed for it. AI systems accumulate in organizations faster than governance frameworks evolve to cover them. A use case that started as an internal productivity tool is now influencing hiring decisions. A model deployed for one market is being used in another where the regulatory context is different. A vendor has updated an underlying model and the organization's internally built layer is now operating on a different foundation than it was when it was approved. None of these are necessarily failures. All of them are things the board should know about — and typically doesn't. The three questions it needs to answer What AI systems are making consequential decisions, and do we have a complete inventory? Most organizations do not have a comprehensive inventory of their production AI systems. AI proliferates in ways that other technology doesn't — it's embedded in vendor products, built by business units operating outside central governance, and updated by vendors without explicit notification to the client. The first deliverable of any AI audit is an accurate map of what exists. "Consequential" is a meaningful threshold here. Not every AI system making recommendations needs the same governance treatment. An internal tool that suggests email response drafts is different from a model that scores customer loan applications or determines which job candidates advance to interview. The audit should focus governance energy on decisions that affect customers, employees, or material financial outcomes. Who is accountable when an AI-driven decision is wrong? This is the question that most AI governance documentation fails to answer concretely. Organizations have AI ethics policies, responsible AI frameworks, and model risk management guidelines. Very few of them name a specific person who is accountable for a specific model's outputs in production. The audit should resolve this to a named individual for each consequential AI system. Not a team. Not a committee. A person, in a role, with defined responsibilities for performance monitoring, incident escalation, and the decision to pause or decommission the system. Could anyone explain a specific bad outcome if required to? This is the forensics question. If a customer was denied credit by an AI model and files a complaint, can the organization trace the specific inputs that drove the decision, explain why the model weighted those inputs the way it did, and demonstrate that the decision was consistent with the model's approved use case and the organization's stated policies? In many organizations, the honest answer is no. The model exists in production, but the audit trail, explainability layer, and documentation necessary to reconstruct a specific decision either don't exist or aren't maintained in a format accessible to anyone outside the technical team. Why internal audit isn't equipped to run it alone Internal audit functions have the independence and mandate to commission this work. They typically don't have the domain expertise to execute it without specialist support — and that's worth being explicit about rather than papering over. An internal auditor assessing whether AI governance documentation is complete can do that independently. An internal auditor assessing whether the documentation reflects what's actually happening in production models, whether monitoring thresholds are set appropriately for the use case, or whether a model's training data is representative of the population it's scoring — that requires someone with operational AI experience. The practical answer is a co-sourced approach: internal audit drives the process and maintains ownership of findings, specialist external support provides the domain expertise for the technical evaluation components. The independence of the finding sits with internal audit. The technical credibility sits with the specialist. This is how most mature compliance functions handle domains where internal expertise is thin — it's not a novel structure, just one that AI hasn't yet been systematically included in. The business case Boards sometimes resist commissioning audits because the output is uncertain and the cost is visible. The AI audit case is stronger than that framing suggests. Regulatory exposure is real and increasing. The EU AI Act creates conformity assessment requirements for high-risk AI systems. Sector-specific AI guidance from financial regulators, the FDA, and employment regulators creates audit trails that organizations will need to produce. An AI audit conducted proactively is a fraction of the cost of a regulatory examination that finds governance gaps the organization didn't know it had. Operational risk is also material. A model that has been quietly degrading for months is a liability that doesn't appear on anyone's radar until it affects enough decisions to produce visible business consequences — customer complaints, adverse outcomes at scale, regulatory notice. An audit that finds this early is worth more than its cost. The D&O angle is worth raising directly with board members. Directors who approve AI strategies and investments are making decisions they will be held accountable for if something goes wrong at scale. An independent, documented review of whether the AI governance is adequate is meaningful protection. Approving an AI investment without it is a risk that sits with the individual director, not just the organization. Frequency and triggers For organizations with material AI exposure — models in production affecting customers, employees, or revenue at volume — an AI strategic audit should be an annual activity, structured similarly to other assurance reviews. Out-of-cycle triggers worth defining: a significant change to a production AI system or its underlying model; entry into a new market or use case with AI involvement; a regulatory examination or enforcement action involving AI anywhere in the industry; a visible AI failure in a comparable organization that prompts questions about whether a similar pattern exists internally; and any M&A that brings new AI systems into the organization. The audit doesn't need to be comprehensive every year. A rolling program that covers the highest-risk systems annually and lower-risk systems on a longer cycle is a practical approach for large organizations with many AI deployments. What it should never be is one-time. The AI landscape inside an organization changes faster than any other technology domain. A clean finding from two years ago is not evidence of a clean position today.

Read full article
Regulatory Exposure Your Legal Team Hasn't Priced In Yet

Regulatory Exposure Your Legal Team Hasn't Priced In Yet

Most enterprise legal teams have a mental model for new regulation built from the GDPR experience: wait for the law to come into force, watch the early enforcement actions to understand where the real exposure sits, then document accordingly. Move fast on the documentation, slow on the underlying change. Find out where the lines are before you invest in compliance infrastructure. That model worked reasonably well for GDPR — enforcement was slow, penalties in the early years were manageable, and the documentation-first approach bought time without serious consequences in most cases. It is the wrong model for the EU AI Act. The difference is structural. GDPR primarily required documentation of existing practices and some adjustments to data handling procedures. The EU AI Act, in its high-risk provisions, requires conformity assessment before deployment — not documentation of what you're already doing, but evidence that the system meets requirements before it goes live. Organizations that apply the GDPR mental model will find themselves with AI systems in production that haven't been through the required assessments, with no clean path to retroactive compliance, and with exposure that compounds with every decision the system makes while out of conformity. What the Act actually says The EU AI Act classifies AI systems into four tiers based on the risk of harm their deployment creates. Unacceptable risk systems are prohibited outright: social scoring by governments, real-time biometric identification in public spaces for law enforcement (with narrow exceptions), AI that exploits vulnerable groups, and systems that manipulate behavior through subliminal techniques. These are banned — no compliance path, no exemption. High-risk systems are the category most enterprises need to focus on. These require conformity assessment before deployment, ongoing monitoring in production, mandatory human oversight mechanisms, detailed technical documentation, and registration in an EU database before deployment. The high-risk categories include: AI systems used in hiring and employee management, credit scoring and credit access, insurance underwriting, educational admission and assessment, law enforcement support, migration and asylum processing, administration of justice, critical infrastructure management, and certain medical device software. Limited risk systems — primarily chatbots and AI-generated content — require transparency disclosures: users must be told they're interacting with AI. Minimal risk systems have no mandatory requirements under the Act, though voluntary codes of conduct may apply. The high-risk category is where most enterprise exposure sits, and it's larger than most legal teams initially assume when they scan the definition. Which enterprise use cases are actually high-risk The hiring dimension alone is significant. Any AI system used to sort, screen, rank, or make recommendations about job candidates or existing employees falls into high-risk. That includes resume screening tools, interview analysis software, performance management AI, and automated scheduling or task assignment systems that affect working conditions. Most large enterprises now use at least one tool in this category, often embedded in HR software they didn't build and may not have evaluated from an AI Act perspective. Credit and financial services exposure is equally broad. AI systems used in credit scoring, creditworthiness assessment, and insurance risk pricing are high-risk. This includes systems used by banks, insurers, and any firm offering financial products that involves AI-driven eligibility or pricing decisions. The medical device software provisions catch more organizations than expected. AI software that is a safety component of a medical device, or AI used to make clinical decisions affecting patient care, falls into high-risk — including software used by healthcare providers, not just device manufacturers. The critical infrastructure category covers AI used in the management of roads, railways, airports, water, gas, electricity, and certain digital infrastructure. This is relevant not just for utilities but for logistics companies, transportation operators, and cloud infrastructure providers. Why the GDPR parallel breaks down Under GDPR, the primary obligation is to document how you process personal data and to meet certain data subject rights requirements. The documentation has to be accurate, but the underlying processing can largely continue while you get the documentation in order. Under the EU AI Act, high-risk systems cannot be deployed until they have passed a conformity assessment. The assessment covers: risk management processes, data governance for training data, technical documentation of system design and performance, logging and monitoring capabilities, transparency and instruction requirements, human oversight mechanisms, and accuracy and robustness requirements. This isn't a documentation exercise that follows deployment. It's a pre-deployment gate. An organization that deploys a high-risk AI system without completing conformity assessment is not behind on compliance documentation — it is operating an illegal system. The penalty structure reflects this. Maximum penalties for prohibited systems are €35 million or 7% of global annual turnover. For high-risk system violations, the maximum is €15 million or 3% of global turnover. These are not late-documentation penalties — they're deployment penalties, applying to every period the non-compliant system was in operation. The extraterritorial reach The EU AI Act applies to any provider that places AI systems on the EU market or puts them into service in the EU, and to any deployer that uses AI systems in the EU — regardless of where the provider or deployer is based. A US company selling software that includes an AI hiring tool to European customers is a provider subject to the Act. A UK company using an AI credit scoring system for its European customers after Brexit is a deployer subject to the Act. A global company running an AI performance management system for its European employees is subject to the Act for those deployments. The extraterritorial scope is broader than most non-EU legal teams initially assume, and the relevant analysis is not "are we an EU company" but "are we deploying AI systems that affect people in the EU." The foundation model provisions The EU AI Act includes specific provisions for general-purpose AI models — what the Act calls GPAI models — which are relevant for any organization using large language model APIs from providers like OpenAI, Anthropic, Google, or Meta. GPAI providers have their own compliance obligations. But organizations that build applications on top of GPAI models are deployers of those capabilities, and the Act creates a liability chain. If you build a high-risk application on top of a GPAI model, your conformity assessment needs to address the GPAI component — you can't simply defer to the model provider's compliance documentation as if your deployment decisions are irrelevant. The practical implication: if you're using an LLM API to power a system that falls into a high-risk use case — an AI system that helps make hiring decisions, credit assessments, or medical triage — the fact that you're using a third-party model doesn't eliminate your conformity assessment obligation for the application layer you've built. What to do now vs. what can wait The Act has a phased implementation timeline. Prohibited system provisions are already in force. High-risk system requirements apply progressively based on system type, with most provisions applying from August 2026 onward, and some extended timelines for specific categories. This creates a window — but it's narrower than it appears, because conformity assessment for complex AI systems takes time. Running the assessment, remediating gaps, completing documentation, and registering in the EU AI database is a 6–12 month process for a well-prepared organization. Organizations that start this process in mid-2026 may find themselves past the deadline. What to do now: inventory every AI system used by your European employees or affecting your European customers. Classify each against the Act's risk tiers. For high-risk systems, begin the conformity assessment process. Flag the GPAI component for any application built on LLM APIs. What can wait: the transparency and labeling requirements for limited-risk systems can be addressed in a second wave. The voluntary codes of conduct for minimal-risk systems are not urgent. The documentation maintenance requirements for compliant high-risk systems are ongoing, not front-loaded. The organizations that will be in the most difficult position in 2026 are those that decided to monitor the situation rather than act on it. The monitoring strategy works when compliance is about documentation. It fails when compliance is a deployment gate — and for the EU AI Act's high-risk provisions, that's exactly what it is.

Read full article