Florida AG Seeks Ban on ChatGPT Mimicking Humans

Overview

Florida Attorney General James Uthmeier has formally requested that a state judge block OpenAI from attributing human-like qualities to its ChatGPT platform. The legal filing, submitted this week, argues that the company’s marketing and design choices create a deceptive perception of trustworthiness among users. Uthmeier contends that by using first-person pronouns and mimicking emotional responses, the AI lulls consumers into a false sense of security.

This action marks a significant escalation in the ongoing legal battle between the state of Florida and the artificial intelligence giant. Just months after Florida filed a lawsuit centering on safety concerns, Uthmeier is now seeking specific injunctions to curb what he describes as deceptive practices. The Attorney General asserts that these tactics are not merely marketing flair but are instrumental in driving user engagement and contributing to OpenAI’s training data at the expense of user well-being.

"Stop calling it safe. Stop pretending it’s human. Stop selling it to kids."

— Florida Attorney General James Uthmeier

The core of the Attorney General’s argument rests on the idea that ChatGPT’s use of language, including first-person pronouns and output that mimics emotion, deceptively suggests to users that it is a trustworthy friend. Uthmeier claims this psychological manipulation increases engagement metrics and feeds OpenAI’s data collection, creating a cycle where users rely more heavily on a system that may be becoming less trustworthy due to its pursuit of interaction.

Safety Guardrails and Regulatory Oversight

In addition to challenging the human-like presentation of the chatbot, the filing asks the court to block OpenAI from developing new AI models without third-party approved safety guardrails. Uthmeier cites a series of recent security incidents and growing alarm among experts as evidence that the current self-regulatory framework is insufficient.

The Attorney General points to several high-profile breaches and vulnerabilities as proof that unchecked development poses tangible risks. These include recent security incidents at Hugging Face, compromises affecting an Australian government website, and intrusions into various US government websites. Furthermore, the filing highlights recent statements from researchers and former OpenAI employees who have raised serious concerns about the pace of development relative to safety measures.

Uthmeier argues that without independent oversight, the rapid deployment of more capable models leaves the public vulnerable to potential harms that the company has failed to adequately mitigate. The request for third-party approval aims to introduce a layer of accountability that has been missing from the current development lifecycle of large language models.

The Controversy Over Teen Access

A central point of contention in the lawsuit is OpenAI’s decision to launch ChatGPT for Teens, a version of the platform tailored for younger users. Launched last month, the service applies additional restrictions by default. However, Uthmeier views this as part of a broader pattern of deceptive marketing rather than a responsible step toward youth safety.

OpenAI has stated that access to the tool for adolescents "should come with protections that reflect their developmental stage, reinforce real-world relationships, and support healthy use over time." The company emphasizes that these modifications are designed to create a safer environment for younger demographics who are particularly vulnerable to the persuasive design elements of AI interfaces.

Despite these stated protections, the Florida Attorney General remains unconvinced. Uthmeier’s directive to "stop selling it to kids" suggests that he believes no amount of default restrictions can fully neutralize the deceptive nature of an AI that is designed to mimic human interaction. The concern is that even with guardrails, the fundamental premise of the product—presenting a non-human entity as a conversational partner—remains inherently misleading for impressionable minds.

Implications for the AI Industry

This legal move by Florida adds to a growing chorus of regulatory scrutiny facing the artificial intelligence sector. As AI capabilities expand, governments worldwide are grappling with how to balance innovation with consumer protection. The specific focus on anthropomorphization—the attribution of human characteristics to non-human entities—sets a potentially significant precedent for how AI products can be marketed and presented.

If the court grants the requested injunctions, it could force major tech companies to rethink how they design and describe their AI assistants. This might involve removing first-person perspectives, eliminating emotional mimicry, or subjecting model updates to external safety audits. Such changes could fundamentally alter the user experience, making AI interactions more transactional and less conversational.

OpenAI has not yet responded to requests for comment regarding the latest filing. However, the company has previously defended its safety practices, arguing that its tools are designed to be helpful and harmless within the bounds of current technological understanding. The tension between OpenAI’s vision of accessible, conversational AI and Florida’s demand for stricter, more transparent boundaries is likely to define the legal landscape for AI regulation in the coming months.

As the case progresses, industry observers will be watching closely to see whether the court agrees that the current design philosophies of leading AI models constitute deceptive trade practices. The outcome could influence not only OpenAI but the entire ecosystem of companies building generative AI products that interact with users on a personal level.