The AI Confidence Trap: Kiwis Claim They Can Spot Fakes

Overview

A stark divide has emerged between New Zealanders’ self-perception and their actual digital behaviour regarding artificial intelligence, according to new cybersecurity research. While a significant majority of Kiwis believe they possess the discernment to identify AI-generated fakes, their online habits tell a troublingly different story.

The 2026 Safe Tech Report by cybersecurity firm Avast reveals that 66 percent of New Zealand adults are confident they would spot a deepfake or AI-generated deception. However, this sense of security is not translating into safer practices. Nearly half of respondents admitted to sharing, or seriously considering sharing, sensitive personal or professional information with public AI tools.

This phenomenon, which experts are calling the AI confidence trap, highlights a growing vulnerability in an era where AI scams are becoming increasingly sophisticated and harder to detect than traditional phishing attempts.

The Confidence-Action Gap

The core of the issue lies in a psychological disconnect. People are becoming more aware of the existence of AI fakes, yet they continue to upload documents that contain private data to platforms that may not be secure.

Stephen Kho from Avast explained that while the public understands the basic red flags of a scam—such as urgency or offers that seem too good to be true—they often rationalise their own risky behaviour.

"Because it's so convenient, we are using these publicly hosted AI tools... and we're uploading our sensitive data," Kho told Nine to Noon. "This includes medical records and financial records because they're so helpful in helping us plan, identify, and do diagnosis."

The report found that 40 percent of New Zealanders have shared or considered sharing sensitive information with AI. Specifically:

21 percent have shared or considered sharing a CV.
21 percent have shared or considered sharing a work document.

Experts warn that treating these public AI platforms as confidantes is a critical error. Unlike a human professional bound by legal and ethical obligations, an AI chatbot is merely a software interface on a server.

"If you're not confident about handing it to someone on the high street, then you shouldn't do it to an AI chatbot," Kho advised. "They are not your secret companions or personal experts. These are just AI bots on the internet."

Risks of Data Privacy

Uploading personal documents to generative AI tools carries risks comparable to posting information openly on the internet. There is no guarantee that the data will remain private, nor is there assurance that it won’t be retained for model training or exposed through security breaches.

Kho pointed out that the fine print of most public AI services states that user information can be used and potentially owned by the hosting companies. Furthermore, insecure configurations of AI platforms can lead to data being dumped into publicly accessible databases, which are then traded on the internet or leveraged for targeted scams.

Many of these tools are hosted overseas, primarily in America and China, raising additional concerns about jurisdiction and data protection standards. To mitigate these risks, Avast recommends that users ensure any information shared with AI is fully anonymised. This means removing any private or personal identifiers that could be traced back to the individual.

The Evolution of Scams

Beyond data privacy, AI is fundamentally changing the landscape of cybercrime by making scams more believable and faster to produce. In the past, phishing attempts were often identifiable by poor grammar, spelling errors, or illogical contexts. Modern AI-generated content eliminates these traditional tells.

The quality of fake websites, the realism of mimicry, and the speed of production have all reached a point where casual observation is rarely sufficient to detect a deception.

Kho cited a recent, high-profile hack as a prime example of this new threat level. In the incident, an employee was contacted in the middle of the night for an emergency conference call. On the screen were the CEO and CIO, appearing in a real-time video meeting.

They instructed the employee to execute a secret $20 million transfer related to an undisclosed merger and acquisition. Despite the unusual circumstances, the employee recognised the faces and voices on the call and complied with the request. It was later revealed that the video and audio were sophisticated deepfakes.

"Recently those are all gone," Kho noted. "So the quality, the speed, how real the fake website is... they're all so real now that it's really really difficult to pick out."

Navigating the AI Era

As AI tools become ubiquitous in daily life, cybersecurity experts urge a shift from blind confidence to cautious verification. The ease of use associated with generative AI often masks the potential consequences of data exposure and the increasing potency of social engineering attacks.

To stay safe, experts suggest the following precautions:

Anonymise Data: Before inputting any document into an AI tool, remove names, addresses, financial details, and other personally identifiable information.
Verify Urgency: If an AI-mediated or video-conferenced request for money or action seems urgent or out of character, verify it through a separate, known communication channel.
Read the Fine Print: Understand how public AI platforms handle data privacy and whether user inputs are used for training models.
Maintain Skepticism: Recognise that high technical fidelity in media does not equal authenticity. Deepfakes can now replicate voice and likeness with alarming accuracy.

The AI confidence trap serves as a warning that human intuition is struggling to keep pace with technological advancement. While New Zealanders feel equipped to spot fakes, the reality is that the threats are evolving faster than traditional detection skills. Bridging this gap requires a proactive approach to digital hygiene and a deeper understanding of how AI technologies interact with personal data security.