Showing Posts From

Risk management

Shadow AI: What's Already Running in Your Organization

Shadow AI: What's Already Running in Your Organization

Before any formal AI program exists, before the steering committee has its first meeting, before the CIO has signed off on a single vendor — your employees are already using AI. Not because they are reckless. Because they have work to finish. I have walked into organizations that spent six months building an AI governance framework and discovered, halfway through the engagement, that the finance team had been using a consumer large language model to draft board reports for the past year. The legal team was using a different one to summarize contracts. Neither team thought they were doing anything wrong. They were getting work done faster. That is the shape of shadow AI. It does not announce itself. It does not appear in your IT asset register. It shows up in the gap between the work people need to do and the tools they have been given to do it. What shadow AI actually looks like The term sounds covert. It rarely is. Most shadow AI use is visible if you look for it — people pasting client briefs into ChatGPT, uploading PDFs to summarization tools, running AI writing assistants over strategy documents. Nobody is hiding anything. They just do not think of it as an IT procurement decision. The categories I see most often: Consumer AI assistants used for drafting, summarizing, researching, and explaining technical material. These are usually the first tools employees reach for because they already use them outside work. The friction is zero. AI features embedded in software people already have — writing assistants in productivity suites, AI tools inside project management and communication platforms. These arrive via a product update, not a procurement decision, and most organizations do not notice until they are already in use. Specialist tools for specific functions: AI contract review, AI coding assistants, AI research tools, AI presentation builders. These typically start as a free trial one person tries. Three months later the whole team is using them, usually without telling anyone. The common thread: easy to access, fast to start, and they solve a real problem. Nobody waits for procurement when they have a real problem. The data that leaves when they do Here is the question I put to leadership teams: what data has left your organization in the last 90 days through AI tools? Almost nobody has an answer. Every prompt sent to a third-party AI tool is data that has left the building. Every document uploaded for summarization. Every contract pasted in for analysis. Every email thread fed to an assistant for context. This is not theoretical exposure. It is live, ongoing, and unmeasured. The specific risk depends on the tool. Some consumer AI products train future models on user inputs by default unless the user has explicitly opted out — a setting most enterprise users have never opened. Some retain query data for extended periods for internal product improvement. Some have data residency terms that have no relationship to the organization's regulatory obligations. Most employees have read none of this. What makes shadow AI data exposure different from other shadow IT is the combination of volume and sensitivity. When someone uses an unsanctioned SaaS product, they tend to generate new data inside that system. When someone uses an unsanctioned AI tool, they are typically feeding existing sensitive material — client information, financial projections, internal strategy — into a system with unknown retention, processing, and training terms. A CTO told me once: "We spent a year tightening our cloud storage permissions, and the whole time people were copying strategy documents into a consumer AI chat." He was not exaggerating. He had run the discovery work. The decision quality problem The data risk is one issue. The decision quality risk is a separate one. When employees use AI tools that have not been evaluated or approved, the outputs they receive carry no governance. There is no audit trail of what the model was asked, what it returned, or how that output influenced a decision. Nobody has tested whether the tool performs reliably on the organization's specific data domain. Nobody has checked whether outputs are factually accurate, whether knowledge cutoffs create blind spots, or whether model behavior introduces biases relevant to the use case. I have seen board presentations drafted with consumer AI assistance that contained subtly incorrect market figures — the kind of error that is hard to catch if you are not already deeply familiar with the material. I have seen contract summaries that missed jurisdiction-specific clauses because the model lacked coverage of that legal context. None of these produced immediate disasters. But they were invisible errors that reached decision-makers before anyone thought to check the source. The problem is not that the tools are necessarily unreliable. The problem is that nobody defined what "reliable enough" looks like for this use case, nobody validated that the tool clears that bar, and nobody has visibility into which decisions have been shaped by which tools. Why a policy does not solve this on its own Every organization that discovers shadow AI responds the same way: they draft a policy. Employees must not use AI tools without prior approval. All AI tools must go through procurement. No sensitive data should be uploaded to external AI systems. These policies are not wrong. They are just insufficient on their own. A policy without detection capability is a statement of intent, not a control. If you cannot observe what tools are in use, you cannot enforce anything. And the detection problem is real — consumer AI tools operate over standard encrypted web traffic that looks indistinguishable from any other browser activity on most network monitoring setups. A policy without a usable alternative is a speed bump, not a barrier. If employees are turning to shadow AI because the approved tooling is slow, limited, expensive, or simply does not exist yet, a policy telling them to stop will reduce usage briefly and then have no effect. People optimize for their work. A policy without a clear explanation of why tends to generate resentment rather than behavior change. If employees do not understand what the actual risk is — not just that "data security is important" but specifically what could happen to the specific data they are handling — they will weigh an abstract policy against a concrete productivity benefit and find the policy unconvincing. What actually works There are four interventions that change the situation. Not instead of policy — alongside it. Run a discovery exercise before making any decisions. You need to know what is actually in use before designing controls. This means endpoint monitoring, network traffic analysis, and honest conversations with department heads. Expect surprises. The goal is not to catch anyone — it is to understand your real exposure before you design a response to it. Move quickly on a sanctioned alternative. The fastest way to reduce shadow AI use is to provide a better approved option. This does not require the best enterprise AI platform with a six-month procurement timeline. It means the minimum viable approved tool that addresses the main use cases driving shadow adoption — often that is simply a properly configured, privacy-compliant version of the same tool people are already using. Create a fast path for new tool requests. The reason shadow AI persists is that the formal route takes too long. Teams wait months for IT to evaluate a tool they need now. Make the process faster and more transparent. Most requests should get a decision within two weeks. The ones that cannot should at least get a clear explanation of why not. Treat the people using shadow AI as a signal. They are telling you where your official tooling is falling short. Employees using unauthorized AI tools are often the highest performers trying to work better. Treating them as a compliance problem to be managed misreads what is happening. Their behavior is product feedback. What to take from thisAudit shadow AI use before designing governance for it. You need to know what is already running before you build controls around it. Consumer AI data terms are not written for enterprise compliance. Read them — specifically the sections on input retention, training use, and data residency — before employees continue uploading sensitive material. A policy without detection is not a control. Invest in observability first, then communicate the policy. The fastest fix is a sanctioned alternative that works. Prohibition without substitution creates resentment, not compliance. The employees using shadow AI are showing you where your approved tooling has gaps. Use that information when planning what to procure next.The organizations I see handle this well are not the ones that moved fastest to write a policy. They are the ones that ran the discovery work, understood their actual exposure, and moved quickly to close the gap between what employees needed and what they were officially allowed to use. That gap is where shadow AI lives.

Read full article
When AI Gets It Wrong at Scale: Incident Response for Executive Teams

When AI Gets It Wrong at Scale: Incident Response for Executive Teams

Most enterprise AI incident response planning amounts to "we will deal with it when it happens." Sometimes that is stated explicitly. More often it is implicit — the program plan does not include an incident response section, the governance framework describes oversight without describing what happens when oversight catches something wrong, and the executive team has not run a tabletop exercise that covers AI failure modes. The absence of preparation is not reckless. It reflects where attention goes when building an AI program: toward capability, delivery, and adoption. Incident response feels like a problem for later. It becomes a problem for now at the worst possible time. AI incidents are different enough from traditional software incidents to warrant explicit preparation. The failure modes are different. The scope assessment is harder. The communications are more complex. And the pressure to continue operating while containing the incident — because the AI system may be deeply embedded in workflows that cannot simply be paused — creates tradeoffs that need to be worked out in advance. Here is what the first 72 hours of an AI incident should look like for an executive team that has done the preparation. The AI incident categories that require executive involvement Not every AI malfunction requires executive attention. A model performance degradation that is caught by monitoring, addressed by the AI team, and resolved within hours without affecting users or external parties is an operational incident, not an executive incident. The categories that require executive involvement are those where the impact exceeds what the AI team can contain operationally: Output failures at scale. The AI system produces incorrect, harmful, or biased outputs that have been acted on by a significant population of users, customers, or decision-makers before the problem is identified. The scope of the downstream impact is not immediately clear. Data exposure. The AI system has caused data to be accessed, transmitted, or disclosed in ways that exceed what was authorized — whether through a prompt injection attack, a misconfiguration, or a vendor incident. The regulatory notification question is immediately live. Regulatory or legal trigger. An AI system output has been identified as potentially discriminatory, as having violated an applicable regulation, or as the subject of a legal claim. The legal and regulatory response needs to begin immediately. Reputational incidents. An AI system failure has become externally visible — through media coverage, customer complaints reaching public forums, or regulatory inquiry. The communications response is time-sensitive. Each of these requires a different primary response, and each has a different set of executives in the lead role. The incident response plan should specify who is in the lead for each category, not assume that a single generic incident response structure covers all of them. Hours 0 to 4: initial assessment The first hours of any significant AI incident involve two parallel tracks: containment and assessment. These need to happen simultaneously, because the decision to maintain, restrict, or shut down the AI system needs to be made quickly and on the basis of the best available information. Containment decision. The immediate question is whether the AI system should continue operating. The options range from full shutdown through restricted operation (limiting to lower-risk use cases) to continued operation with enhanced monitoring. The decision depends on the nature of the incident, the criticality of the AI system to business operations, and the risk of further harm if operation continues. This decision needs to be made by the executive sponsor with input from the CTO and, where relevant, the CISO and general counsel. It should not default to the AI team alone, because the business continuity implications extend beyond the technical assessment. Scope assessment initiation. What has the AI system done, to whom, and with what data? The scope assessment for an AI incident is harder than for a traditional data breach because AI systems do not maintain the same kind of access logs that database systems do. The query and output logs of the AI system are the starting point. Reconstructing the impact from those logs requires AI-specific expertise. Assign the scope assessment as a named responsibility to the CTO's function. Set a first checkpoint time — four hours or less for an initial estimate of scope — and a full report timeline. Hours 4 to 24: assessment and notification decisions By the end of the first 24 hours, the executive team needs to have answered several questions that drive subsequent decisions. Regulatory notification. If the incident involves personal data, has the assessment produced enough clarity to determine whether the incident triggers regulatory notification obligations? In most jurisdictions, the notification clock starts at discovery of a breach, and the notification period is 72 hours in many regulatory regimes. This timeline is unforgiving. If the incident involves personal data and there is a reasonable possibility of notification obligation, engage the data protection officer and external legal counsel immediately — do not wait for full scope clarity. Customer and partner notification. Separate from regulatory obligations, does the incident require proactive notification to affected customers or partners? This decision involves commercial judgment as well as regulatory judgment. The general counsel and the CTO together need to produce a recommendation for the executive sponsor. Internal communications. Who inside the organization needs to know about the incident, at what level of detail, and at what stage of the investigation? The communications approach should be deliberate — too narrow creates governance failures; too broad before facts are established creates confusion and external leak risk. Hours 24 to 72: remediation and external response By 48 hours, the executive team should have enough understanding of the incident to make the decisions that need to be made. Remediation plan. What is the AI team doing to fix the underlying problem, what is the timeline, and what assurance does the business have that the fix is complete before the system returns to full operation? The AI team provides the technical answer; the executive team validates that the assurance standard is sufficient. External communications. For incidents that have external visibility — regulatory inquiry, media coverage, significant customer impact — the external communications response needs to be coordinated. The communications team needs to work from accurate, verified information. Premature statements that are later contradicted damage credibility significantly. Regulatory response. If a regulatory notification has been made, assign the regulatory liaison role clearly. The communications with regulators should be managed through a single channel, should be accurate and complete, and should not outrun the organization's actual knowledge of the incident. Building the plan before you need it An incident response plan that is written during an incident is worse than one written in advance. The decisions about who leads which type of incident, what the shutdown criteria are, what the notification thresholds are, and who the external legal and regulatory advisors are need to be made without the time pressure and reputational stakes that an active incident creates. Specifically: run an AI incident tabletop exercise before the first significant AI system goes live. The exercise should cover at least one of the major incident categories — scale output failure, data exposure, regulatory trigger — and should involve the executive team, not just the AI and security functions. The outputs of the tabletop should be a tested incident response plan that reflects the organization's actual structure and decision-making processes. What to take from thisAI incidents are distinct enough from traditional software incidents to require their own response planning. The scope assessment, containment decision, and regulatory notification analysis all require AI-specific expertise and approaches. The containment decision — whether to maintain, restrict, or shut down the AI system — needs to be made by the executive sponsor, not delegated to the AI team. The business continuity implications require that level of authority. For incidents involving personal data, the regulatory notification clock starts at discovery. Engage legal immediately if there is a reasonable possibility of notification obligation — do not wait for full scope clarity. Run a tabletop exercise covering AI incident scenarios before the first significant AI system goes live. Test the plan with the people who will execute it, under conditions that approximate the time pressure of a real incident. External communications for AI incidents need to be coordinated and accurate. Premature statements corrected later are more damaging than a short delay for verification.

Read full article