AI Hallucination Nearly Triggered US Military Action Against China

Overview

According to reports, this spring, US military aircraft took off to carry out armed operation missions targeting a Chinese vessel. However, at the last moment, US officials discovered that the intelligence driving this operation originated from a hallucination of an AI chatbot. This discovery led to the mission being urgently halted, and the United States nearly became involved in a military conflict with China as a result.

CNN reported on this incident on Friday, revealing that as decision-makers increasingly rely on artificial intelligence, errors generated by systems can spread rapidly through the entire chain of command, reaching decision-makers even before they are questioned.

The Birth of False Intelligence

This intelligence report was produced during the war with Iran and claimed that the target vessel was carrying parts used for a nuclear weapons program. In fact, this false intelligence stemmed from a combination of human error and the unreliability of AI.

An analyst from the Special Operations Command used an AI chatbot in an attempt to synthesize open-source data with classified signals intelligence. However, the chatbot misjudged the ship's cargo manifest, generating non-existent nuclear-related information.

• The analyst subsequently used the same tool again to format these erroneous findings into an official-looking summary report.
• The report was then distributed across various command channels and became the basis for launching the strike.

This "close call" incident highlights the fatal risk of lack of oversight in human-machine collaboration within a wartime environment.

The Trade-off Between Speed and Safety

This narrowly avoided conflict occurred against the backdrop of the US military competing to integrate AI to enhance its intensity against China. The Pentagon views AI as a key means to accelerate the kill chain, ensuring that commanders can respond quickly at critical moments.

"It is crucial to understand the inherent uncertainties of large language models," said Jake Steckler, a research scholar at GovAI and a US Army veteran. "But this is especially critical for decisions that could lead to the use of force—such as target targeting, intelligence analysis, or operational planning. These decisions concern life and death."

Steckler pointed out that while this incident should be seen as a warning to increase AI safety measures rather than a reason to abandon the technology, if there is a sole priority given to adopting speed while ignoring all other factors, similar incidents will occur frequently, ultimately damaging personnel's trust in these systems.

This loss of trust could, in turn, fundamentally slow down the long-term deployment of AI in the military domain. How to establish effective human oversight mechanisms while maintaining decision-making speed has become a core challenge urgently requiring resolution for the US military and global defense agencies alike.