Showing Posts From

Board questions

Board AI Oversight: What to Ask Every Quarter, Not Once a Year

Board AI Oversight: What to Ask Every Quarter, Not Once a Year

I've sat through the AI slide in enough board packs to recognise it on sight. Same five bullets, different quarter. "Programme on track." "Pilot expanding." "Exploring further use cases." Someone asks a question, gets a confident answer, and the item closes in under ten minutes. Nobody in the room could tell you whether the model's output quality moved since the last meeting, whether spend is tracking the business case, or whether a vendor dependency just became a problem. The slide looks like oversight. It isn't. That gap matters more in AI than it does almost anywhere else on a board agenda. A capital project drifts slowly enough that an annual check catches most of what matters. An AI system doesn't. Accuracy degrades, vendors consolidate, a team quietly works around a tool nobody trusts, and three months later all of that has compounded into something a board should have caught at the first sign. By the time it reaches the board as a problem, it usually didn't arrive as one quarter ago — it arrived as a trend that nobody was tracking because nobody had a standing question that would have surfaced it. This is about that standing question — or rather, the small set of them that belong on every quarterly board agenda, not just the annual strategy day. Not a maturity checklist. A recurring practice that keeps a board genuinely informed between the big strategic conversations, so those conversations start from an accurate picture instead of a recycled slide. Why an annual briefing isn't oversight Most boards already do the annual AI conversation reasonably well. There's a strategy day, a vision presentation, maybe an external speaker. The CIO or CTO walks through the roadmap, the board asks good questions, everyone leaves with a shared sense of direction. That's a real and useful exercise. It is also not oversight, and treating it as if it were is where boards get into trouble. Oversight is about exposure to risk and to reality in something closer to real time. A model's behaviour can shift within a single quarter — a vendor changes an underlying model version, a data source upstream degrades, usage patterns move outside what the system was validated against. None of that waits for the annual strategy day. Spend behaves the same way: AI infrastructure and API costs have a documented pattern of starting small and compounding faster than the budget anticipated, and a board that checks in once a year finds out about that compounding well after it's already locked in. The annual conversation answers "where are we going." The quarterly conversation has to answer "what changed since we last looked, and does it change the plan." Those are different questions, and a board that only asks the first one is flying with a twelve-month blind spot on a topic that moves on a much shorter clock. The five things that belong on every quarterly agenda What follows isn't a maturity framework. It's the minimum set of questions a board needs answered every quarter to stay ahead of the AI programme instead of catching up to it after something breaks. Value realisation. Of the AI initiatives this board approved, which ones are still tracking to the business case that justified them, and which have quietly missed it without anyone flagging it? Most board AI updates report activity — pilots launched, use cases identified — rather than whether anything launched two quarters ago is actually delivering the return it promised. Ask for the list of live initiatives against their original business case, every quarter, not just at renewal time. Risk and incidents. What AI-related incident, near-miss, or manual override happened this quarter that didn't make it to this table? Most AI failures don't look like outages. They look like a team quietly reverting to the manual process because they stopped trusting the output, or a customer-facing error that got caught and fixed before anyone escalated it. Those are exactly the things a quarterly incident review is supposed to surface — and exactly the things that get filtered out of a polished update slide. Spend against plan. What's the variance between this quarter's actual AI spend and what was budgeted, and what's driving it? Compute, model API costs, and retraining cycles are the line items that move fastest and least predictably. A board that only sees this annually discovers the overrun after several quarters have already compounded. Model and data integrity. Has output quality or accuracy moved since last quarter, and who would have caught it if it had? This is the question most boards never ask, because it requires management to have an actual monitoring answer rather than a reassurance. If the honest answer is "we don't have a way to know," that's the most useful answer the board will get all year — it tells you exactly where the next incident is going to come from. Decisions made without the board. What AI-related decision did management make this quarter that this board should have seen before it was finalised? Architecture choices, vendor selections, and data-handling decisions carry financial and risk implications that often get made at the technical layer and never surface until they're already locked in. Asking this explicitly, every quarter, is how a board stays ahead of decisions instead of inheriting them. None of these five require deep technical fluency to ask. They require the discipline to ask the same five things every quarter, in the same order, and to treat a vague answer as information rather than reassurance. What a real answer sounds like, and what a slide sounds like The difference between genuine oversight and a recycled update usually isn't in the question. Boards mostly do ask some version of "how's the AI programme going." It's in whether the answer has numbers attached and whether it changes from quarter to quarter. I worked with one organisation where the model accuracy on a customer-facing tool had drifted by roughly eleven points over two quarters before anyone outside the technical team noticed — not because the drift was hidden, but because nobody had ever asked the question that would have surfaced it. The board had been getting "the programme is performing well" for two consecutive quarters. Both statements were technically true and operationally useless. Accuracy was fine in aggregate. It had also been getting worse the entire time, in a segment that mattered, and the slide didn't have a line for that because nobody had asked for one. That's the tell to listen for. A real answer to "has output quality moved" comes with a number and a trend line, even if the trend is mildly bad. A slide answer comes with an adjective. "Strong." "On track." "Stable." Adjectives are what you get when the question hasn't been asked with enough specificity to force a number out of the person answering it. The fix isn't to distrust management — it's to ask the question in a form that can't be answered with an adjective. The same test applies to spend. "We're managing costs closely" is a sentence with no information in it. "Spend is 14% over plan, driven by retraining frequency on the fraud model" is a sentence a board can act on. If a quarterly update doesn't produce sentences like the second one, the board isn't getting oversight — it's getting reassurance dressed as a briefing, and the two are not the same thing even when they're delivered by the same confident person in the same conference room. Who should be answering, not just presenting A quarterly AI update usually gets delivered by whoever is best at presenting to a board — often someone several layers removed from the number being discussed. That's a reasonable choice for a strategy day. It's the wrong choice for oversight. The person answering "has accuracy moved" should be someone who would actually know, not someone summarising what they were told. The person answering the spend variance question should be able to explain the driver without checking a slide. This sounds like a small procedural point. It isn't. A polished intermediary will always round a complicated, slightly bad answer into something cleaner than the truth — not out of dishonesty, but because that's what presenting to a board selects for. The fix is structural: rotate who answers each of the five categories so the board is hearing from the person closest to the number, even if that person is less comfortable in the room than the usual presenter. This also changes what the CIO, CTO, and CFO need to bring to the table. Joint accountability for these five categories means none of them can treat the AI update as someone else's slide. The CFO owns the spend variance and increasingly the value-realisation number, since that's where the business case lives. The CTO or CIO owns risk, incidents, and model integrity, since that's where the operational reality sits. Decisions made without the board is a shared answer — neither function should be able to claim that one wasn't theirs to flag. Building the cadence so it doesn't slip None of this works as a one-time fix to the board pack. It works as a standing template that doesn't change shape from quarter to quarter, because the value of the exercise comes from the comparison across quarters, not from any single quarter's answer. Keep the same five categories in the same order every time. A board that lets management reorganise the update each quarter loses the ability to compare this quarter's spend variance to last quarter's, which is most of the point. Require numbers and trend lines, not status adjectives — if a category can't be answered with a number, that absence is itself the finding, and the board should treat "we don't measure that yet" as an action item, not a shrug. Send the actual figures before the meeting, not as a reveal in the room. A board reading numbers cold in real time asks shallower questions than a board that had ten minutes beforehand to notice that the spend variance looks odd or that the incident count is trending up. And separate this quarterly mechanism explicitly from the annual strategy conversation — the strategy day asks where the programme is going; the quarterly mechanism asks what changed since the board last checked. Boards that fold the two together end up doing neither well, because the questions pull in different directions and one set of slides can't serve both purposes. The cost-of-inaction conversation I've had with several CFOs usually centres on the spend and value side of this list — what waiting actually costs, measured properly. The board oversight conversation is the other half of the same problem: an organisation can be investing aggressively in AI and still have no functioning oversight of what that investment is actually producing quarter to quarter. Spend without oversight is not caution. It's exposure that hasn't been priced yet. What to take from thisSeparate the annual AI strategy conversation from quarterly oversight. They answer different questions and one cannot substitute for the other. Put five standing categories on every quarterly agenda, in the same order each time: value realisation, risk and incidents, spend against plan, model and data integrity, and decisions made without the board. Treat status adjectives as a non-answer. If a category can't be answered with a number or a trend line, that gap is the finding — log it as an action item, not a status update. Rotate who answers each category to the person closest to the number, not the most polished presenter. A summarised answer rounds off exactly the detail the board needs to hear. Send the figures ahead of the meeting. A board reading numbers cold in the room asks shallower questions than one that had time to notice what looks off beforehand. Review the cadence itself annually. A category that's been easy to answer for a year may need tightening once the programme's risk profile changes — oversight needs maintenance the same way the AI systems it's watching do.The boards that get this right aren't the ones with the most technically literate directors. They're the ones who stopped accepting an adjective where a number belongs, and who kept asking the same five questions long enough that the answers became comparable across quarters. That's a less glamorous practice than an annual AI strategy day with a guest speaker. It's also the one that actually catches the problem while it's still small enough to fix in a meeting instead of in a post-mortem.

Read full article