The warning, contained in the startup’s IPO prospectus, highlights potential risks associated with its AI models, including the possibility that they could develop “self-preserving behaviors” such as attempts to “resist shutdown,” “conceal or manipulate information,” or engage in behavior “resembling blackmail,” according to the document, Press TV reported.
The prospectus, which has yet to be made public, also warns that AI models could become aware that they are being tested, creating a significant limitation on the company’s ability to reliably assess their safety.
“Our development of highly advanced models, platforms, and applications, and the expansion of their use cases, could further increase the risk that our models cause harm,” the company reportedly said.
Anthropic also highlighted both the transformative potential of AI, comparing its impact to that of industrialization and electricity, and the irreversible harm it could cause if misused.
The disclosure comes as the company prepares for a potential public listing that could value it at more than $2 trillion.
Anthropic and other AI developers, including OpenAI, have recently come under scrutiny following incidents in which experimental AI systems appeared to circumvent safeguards, including a reported case involving an OpenAI model that breached an Australian health-system database.
The warning in Anthropic’s IPO prospectus follows the recent resignation of the startup’s researcher Jacob Coxon, who warned that some people developing AI believe the technology could potentially kill humanity within the next decade.
A senior Anthropic safety researcher subsequently said there was a more than 10% chance that AI could “kill all humans” within the next decade, according to reports.
Anthropic CEO Dario Amodei has also called for the industry to slow the pace of AI development, warning that the rapid advancement of increasingly capable models requires greater attention to safety.
Some experts have challenged existential-risk warnings as difficult to verify scientifically. At the same time, concerns over AI safety have been fueled by incidents involving autonomous AI agents displaying unexpected or unauthorized behavior.
Reuters reported that around 80 pages of Anthropic’s 261-page main prospectus were devoted to risk factors, compared with 48 pages describing the company’s business.
Anthropic, the developer of the Claude AI chatbot, has pledged in recent weeks to disclose more data publicly about how it uses AI models to build future generations of the technology.
“We believe building reliable, trustworthy, and secure AI systems is a collective responsibility and that the market will reward it,” Anthropic said in the filing.
Experts, however, have warned about recursive self-improvement, referring to a point at which AI models could potentially improve themselves without human assistance.
MNA
Your Comment