Your Business Uses AI. What Happens When It Gets Something Wrong?

What changed

On Sunday, September 20, U.S. Treasury Secretary Scott Bessent said the United States has proposed a new "notification mechanism" for artificial intelligence incidents that could affect national security, according to the Associated Press. He made the proposal in talks in New York with Chinese Vice Premier He Lifeng, ahead of a planned meeting between President Trump and Chinese leader Xi Jinping at the White House later this week. In plain terms, the idea is a channel for the two governments to warn each other when an AI incident becomes serious enough to matter for national security.

As of the reports we found, nothing has been agreed. NBC News reports that both sides agreed to keep talking about AI, and that China's official readout said only that the sides had discussed it, without addressing the U.S. proposal.

The honest nuance

This is a proposal, not a working system. Al Jazeera notes that the exact triggers for an alert have not been made public, and no one has described what counts as an incident, who would send an alert, or how fast. It is also a government matter about national security. A wrong answer from your customer-service assistant is a different kind of event, and nothing here says that businesses face the same risks or that any rule for them is coming. We are not going to speculate about how the talks go.

What we are borrowing is only the shape of the idea: agree in advance on what counts as an incident, who gets told, and what happens next, rather than working it out during the incident.

Why it matters for your business

A wrong answer in a chat window is something you notice and fix. A tool connected to your email, customer records, store, books, or ad account can act on a mistake before anyone looks. For a small business, an AI incident is usually mundane: an assistant sends a customer the wrong price or policy, reaches files or customer data it should not have, changes a record or setting nobody approved, keeps making the same bad recommendation, or an integration starts misbehaving after an update.

The risk in the moment is confusion, not the tool itself. Nobody is sure whose call it is to switch it off, where the record of what it did is kept, or whether anyone needs to be told. People improvise, and the usual first instinct, quietly fixing the damage, can erase the evidence you need to understand what happened. Deciding this in advance is much cheaper than deciding it while a customer is on the phone.

What to actually do

For each AI tool that can reach something important, make a one-page incident card. Start with a line on what counts as an incident for that tool (wrong information sent out, data reached that should not have been, a change nobody approved, the same bad output repeated, an integration acting strangely). Then fill in six fields. Owner: one named person, plus a backup. What it can reach: the systems and data, copied from your access list. How to stop it: the exact steps to pause the automation or revoke its connection, and who may do that when the owner is away. Where the logs are: where the activity history lives and how long it is kept, and save a copy before changing anything. Who must be told: the owner, the vendor, any affected customers, and your accountant or attorney if customer data or contracts are involved. What a human checks before it goes back on: the cause, the affected records, and the permissions.

Example, for an AI assistant that drafts and sends replies from your support inbox. Owner: Dana, backup: you. Reach: the support inbox and order lookup. Stop: turn off auto-send, and if that fails, remove its connection in the email account's app settings. Logs: the tool's activity history and the Sent folder, exported first. Tell: you right away, the vendor if it looks like a bug, and each affected customer with a short correction from a person. Before restoring: someone reads the last 20 sent replies, and it runs draft-only for a week. For anything involving customer data or legal duties, ask a qualified professional what you need to report.

Then run a 10-minute what-if drill on one card: "It just emailed 40 customers the wrong refund policy. Who finds out, who turns it off, and where do we look?" If anyone hesitates, fix the card.

From Kindloom Labs

Part of planning for mistakes is knowing what kinds to expect. The free Quick Read What AI Can and Can't Do covers where AI tends to fall short, like facts and judgment, which helps you decide what should always get a human check.

Sources

• Bessent: US proposes AI incident alert system in talks with China (Associated Press, via ABC News)
• U.S. proposes exchanging AI safety alerts with China, Bessent says (NBC News)
• What's the US-China AI 'hotline' that Trump plans to pitch to Xi Jinping? (Al Jazeera)