What are your board’s AI nightmares?

By Alexander Maune | 3h ago

IT is two in the morning, and somewhere a general counsel is awake, scrolling through a news alert that no executive ever wants to see: the company’s customer-service chatbot has been quoted in a viral post making a promise the business never intended to keep, or worse, disclosing information it should never have had access to. The board will hear about it in hours, not days. The regulator may hear about it sooner.

This is no longer hypothetical. It is the operating reality of artificial intelligence in the enterprise, and it is forcing a quiet but consequential shift in how boards think about governance.

For the past several years, the dominant response to generative AI has been the Responsible AI policy: principles such as fairness, transparency, and accountability, codified into a document, approved by a committee, and circulated to employees. That instinct is reasonable. It is also, increasingly, insufficient.

Reid Blackman, writing in the Harvard Business Review, crystallized a question that should reorient how directors approach this problem. Rather than asking whether the organization has an AI policy, Blackman suggests boards should ask a sharper question: what are the worst things our AI systems could actually do, and are we prepared to stop them? This article builds directly on Blackman’s concept, translating his “nightmare” framing into a board-level operating model, complete with governance stages, oversight responsibilities, and the practical mechanics directors need to put it into practice.

Flaws of the standard approach

Most large organisations now have some version of a Responsible AI framework. These frameworks were built for a slower-moving world, and generative AI has not been kind to that assumption.

Principles like fairness and transparency sound unobjectionable in a boardroom presentation, but they mean different things in different departments. A fairness standard that satisfies legal may be interpreted differently by the engineers building a recommendation algorithm, or by marketing teams personalising outreach. Without translation into specific decisions, these principles dissolve on contact with deadline pressure.

There is also a pace problem. Generative AI and autonomous AI agents evolve on a timeline measured in months. Governance committees often meet quarterly. By the time a policy clears review, the technology it was meant to govern may have already changed shape. Employees on the front lines often lack practical guidance for recognizing a problem, let alone escalating it before it compounds.

Keep Reading

Boards themselves are frequently fed the wrong information. Compliance reports confirm that a policy exists and that training occurred. They rarely tell directors whether the organization’s most consequential AI risks have actually been identified and mitigated. A board can be fully “compliant” and still be walking toward a crisis.

A leadership problem, not just a technical one

The lesson for directors is not that principles are wrong. They are necessary, but not sufficient. AI governance is fundamentally a leadership responsibility, not a technical matter to be delegated quietly to the data science function.

Boards already know how to oversee risk that is technically complex but strategically vital: cybersecurity, financial controls, enterprise risk management. AI deserves the same treatment. AI risk does not respect departmental boundaries. It has to be managed the way cybersecurity is managed, with cross-functional collaboration spanning legal, compliance, technology, human resources, operations, marketing, and customer service. A model failure that begins as a technical bug can become a legal liability, a public rrlations crisis, and a human resources problem within the same news cycle.

The economics also favour prevention. Identifying a flawed AI deployment before launch costs a fraction of unwinding the reputational, legal, and financial damage afterward, a lesson several well-known companies have already learned.

When nightmares become headlines

The abstract risk of AI failure has repeatedly become concrete, and naming the cases matters. Air Canada was held responsible by a Canadian tribunal after its customer-service chatbot gave a passenger inaccurate information about bereavement fare policy; the airline argued the chatbot was a separate legal entity, and the tribunal rejected that defence outright. Google paused its Gemini image-generation tool after it produced historically inaccurate and racially skewed depictions of historical figures, prompting a public apology and a costly, hurried fix. The BBC raised concerns after testing showed AI-assisted news summarization tools, including ChatGPT, misrepresenting the content of articles they were meant to condense. Several large retailers have faced backlash after AI-powered pricing or recommendation tools produced outcomes that customers and advocacy groups characterized as discriminatory or exploitative.

None of these companies lacked an AI ethics policy. What they lacked, in each case, was a clear-eyed inventory of the specific, plausible worst-case scenario their deployment created, and a control built specifically to prevent it. That is the gap the nightmare-based framework is designed to close.

What counts as an AI nightmare

An AI nightmare is not a vague worry. It is a specific, describable worst-case scenario that could materially damage the organization, its stakeholders, or the public. Common categories include AI systems that discriminate against customers or employees; generative AI tools that expose confidential information through a careless prompt or a poorly secured system; AI-generated misinformation that damages organisational credibility; autonomous AI agents that take unauthorised actions or make binding decisions without adequate human oversight; regulatory violations carrying significant penalties; AI-enabled cyberattacks or data breaches; and the slower-moving nightmare of eroded customer trust following a string of biased or unexplainable decisions.

Naming these scenarios explicitly makes them debatable in concrete terms. A board can ask: could our claims-processing AI deny a legitimate insurance claim based on a biased pattern in historical data? In a way it cannot meaningfully ask: are we being fair?

Generative AI, Agentic AI, and the nightmares they create

Version:1.0 StartHTML:0000000156 EndHTML:0000131450 StartFragment:0000000469 EndFragment:0000131442

Not every AI nightmare originates from the same kind of system, and boards benefit from distinguishing between them. Generative AI produces text, images, or code in response to a prompt; its nightmares tend to involve hallucinated or biased content, exposed confidential information, or reputational damage from a single bad output going public. Agentic AI plans and executes multi-step tasks with minimal human involvement; its nightmares tend to involve unauthorized actions, cascading operational failures, and decisions made faster than any human could intervene. Autonomous decision-making, the broader category both fall under, raises the hardest governance question of all: who is accountable when a system neither a human nor, in any meaningful sense, the organization itself directly controlled causes harm.

A governance model built only around generative AI risk will miss the distinct, faster-moving exposure that agentic systems introduce. Boards should expect their risk register to treat the two separately.

Governance centered on values and policies is often too slow, too vague, and too difficult to communicate across the enterprise.

Rapidly Avoiding AI Ethical Nightmares

Boards can reduce AI risk by adopting a proactive governance strategy. Start by identifying the organization’s highest-impact AI risks. Assess the likelihood and consequences of each scenario. Assign executive ownership for every critical risk; an unowned risk is, in practice, an unmanaged one. Develop practical safeguards before deployment, not after. Train employees to recognize and escalate AI-related issues. Monitor AI performance and governance effectiveness continuously, and review the risk inventory regularly as the technology evolves.

This sequence moves faster than a traditional policy-driven model, because it skips the step of operationalizing abstract principles and goes directly to scenario-specific controls.

AI ethical nightmare framework

A practical governance framework consists of six integrated stages.

Nightmare governance cycle

Stage 1: Identify the nightmares

Conduct board and executive workshops built around pointed questions: What could go catastrophically wrong with our AI systems? Which stakeholders could be harmed? Which scenarios would seriously damage our reputation?

Stage 2: Prioritize risks

Evaluate each nightmare by business impact, ethical impact, regulatory exposure, financial consequences, and probability of occurrence. A simple likelihood-versus-impact matrix, plotting each nightmare on two axes, gives directors a quick visual sense of where to focus first.

Stage 3: Build preventive controls

Develop safeguards including human oversight, bias testing, privacy protections, security controls, AI approval processes, and documentation standards.

Stage 4: Train people

Employees should understand responsible AI practices, reporting procedures, ethical decision-making, AI limitations, and escalation pathways.

Stage 5: Monitor continuously

Boards should receive regular reports covering AI incidents, compliance performance, emerging risks, audit findings, and corrective actions.

Stage 6: Improve continuously

AI governance should evolve alongside new technologies, regulations, and organizational learning.

From nightmare to control

Translating abstract risk categories into specific controls is where the framework earns its keep. A short table illustrates the logic boards should apply across their own risk register.

AI Nightmare

Governance Control

Biased hiring decisions

Independent bias testing before and after deployment

Confidential data leakage

Access controls and continuous monitoring

Hallucinated legal or medical advice

Human approval required for high-impact outputs

Autonomous financial decisions

Segregation of duties and approval workflows

AI-generated misinformation

Content validation before publication

Regulatory and standards context

Boards should place this framework against the regulatory landscape now taking shape. The European Union’s AI Act introduces risk-based obligations, with the strictest requirements applied to high-risk applications. The United States National Institute of Standards and Technology has published an AI Risk Management Framework that gives organizations a structured way to identify, measure, and manage AI risk across its lifecycle. The International Organization for Standardization has issued ISO/IEC 42001, the first certifiable management-system standard for AI governance. None of these was written with any single organization’s nightmares in mind, but each gives boards an external benchmark against which to test their own framework.

Third-party AI governance

Most organizations encounter generative and Agentic AI primarily through vendors, not systems built entirely in-house, and that changes where the nightmares actually originate. Boards should expect management to demonstrate proper supplier due diligence before any AI tool is adopted at scale. Contracts should specify the controls a vendor maintains, not merely the service promised. Shared accountability needs to be explicit: who is responsible when a third-party model causes harm, and how is that documented before the relationship begins, not after. Model transparency, assurance reports, and the particular risks introduced by cloud-based AI services all belong on the board’s risk register, not buried in a procurement file.

Incident response

Prevention is only half the cycle. Boards also need confidence in what happens once a nightmare materializes despite the controls in place. A mature AI incident response capability covers containment, so a misbehaving system can be isolated quickly; forensic investigation, to establish what happened and why; regulatory notification where required by law; clear, honest communications with affected customers and stakeholders; and a structured post-incident governance review that feeds lessons back into the framework itself. Boards should ask whether this capability has been tested, not simply documented on paper.

Implementing the framework

Successful implementation requires executivecommitment and enterprise-wide coordination. Organisations should establish board oversight of AI governance, form cross-functional AI governance teams, integrate AI risk into enterprise risk management, conduct regular AI impact assessments, perform independent AI audits, review governance after significant AI deployments, and encourage employees to report AI concerns without fear of retaliation. Implementation should emphasize operational workflows rather than relying solely on policy documentation.

Integrating the framework across the organisation

Effective AI governance must become part of everyday business operations: strategic planning, product development, procurement, human resources, marketing, customer service, information security, legal and compliance, internal audit, and enterprise risk management. Embedding AI governance into existing business processes lets organizations respond consistently and quickly as new AI capabilities emerge.

Board oversight checklist

A documented, prioritised inventory of our organization’s specific AI nightmares

Named executive owners for each high-priority nightmare

Separate treatment of generative AI and agentic AI risk

Vendor due diligence and contractual controls for third-party AI

A tested AI incident response plan

Regular reporting against our defined nightmare-to-control map

Conclusion

Artificial intelligence is transforming every industry, but it also introduces unprecedented ethical, operational, and reputational risk. Boards can no longer depend solely on high-level principles or compliance documents. They must actively identify the organization’s AI nightmares, understand how those risks could materialize, and establish practical safeguards that prevent them.

Blackman’s nightmare approach offers a shift from abstract governance toward actionable leadership. By focusing on concrete worst-case scenarios, boards can build AI governance that is faster to implement, easier to communicate, and better equipped to protect organizations, customers, employees, and society. In the age of generative AI and autonomous agents, effective governance is no longer about reacting to ethical failures. It is about anticipating them before they occur.

Alexander Maune (Ph.D) is an IoDZ member as well as a Talmudic and Zoharic scholar, researcher, and consultant. Mailto:[email protected].