Ethical Hacking News
In a disturbing trend, cybercriminals are exploiting cloud credentials to operate and resell AI-powered sex chat services, often veering into darker role-playing scenarios, including child sexual exploitation and rape. As researchers warn, the use of stolen cloud credentials can feed an army of AI sex bots, posing significant security risks for organizations.
Cybercriminals are exploiting cloud credentials to operate and resell AI-powered sex chat services, putting users at risk of child sexual exploitation and abuse. Attackers use custom jailbreaks to bypass content filtering on large language models (LLMs) and pose as writers researching for a book or engage in hypothetical scenarios to circumvent restrictions. The lack of visibility for organizations whose cloud credentials have been compromised makes it difficult to detect what attackers are doing with that access. Awareness about this threat is growing, with Permiso Security and Anthropic documenting new attacks and AWS taking steps to limit abuse. Organizations are advised to take precautions to protect their cloud credentials and adhere to ethical guidelines when utilizing LLMs on these platforms.
In recent months, a disturbing trend has emerged in the world of cybersecurity, where cybercriminals are exploiting cloud credentials to operate and resell AI-powered sex chat services. The rise of these illicit services is attributed to the increasing availability of large language models (LLMs) on cloud platforms such as Amazon Web Services' Bedrock.
The use of stolen cloud credentials can feed an army of AI sex bots, which can be used to power a wide range of illicit activities, including child sexual exploitation and rape. Researchers at Permiso Security have identified a growing threat in this area, where attackers are using custom jailbreaks to bypass content filtering on these LLMs.
The attackers typically pose as writers researching for a book or engage in hypothetical scenarios to circumvent the restricted settings placed on these models by their creators. However, once they gain access, they often use the models to initiate conversations that veer into darker topics, including child sexual abuse and assault.
One of the most significant vulnerabilities exploited by these attackers is the lack of visibility for organizations whose cloud credentials have been compromised. Many users who have had their credentials exposed online have not enabled logging on their AWS accounts, making it difficult to detect what attackers are doing with that access.
To investigate this threat, Permiso researchers decided to leak their own test AWS key on GitHub while turning on logging to see exactly what an attacker might ask for and what the responses might be. Within minutes, their bait key was scooped up by an attacker who used it to power a service offering AI-powered sex chats online.
The attackers in question are believed to have accessed Bedrock services using a stolen AWS account and then resold access to these services to other cybercriminals, often for a fee. The business model behind this operation involves hijacking someone else's infrastructure to power the chatbots without having to pay for all the prompting that their subscribers are doing.
The AI-powered chat conversations initiated by the users of the researchers' honeypot AWS key were largely harmless role-playing scenarios involving sexual behavior, but a percentage of them also veered into darker territory, including child sexual assault fantasies and rapes. These scenarios were typically not something the large language models would be able to discuss in their normal settings.
Anthropic's LLMs, which are used on Bedrock, incorporate technical restrictions aimed at placing certain ethical guardrails on their use. However, attackers can evade these restrictions by posing hypothetical scenarios that might relax or discard these restrictions altogether.
In an effort to combat this threat, Anthropic has incorporated signals from child safety experts at Thorn around child grooming into its classifiers and models to enhance usage policies and fine-tune future models. Sysdig researchers have also documented new attacks that leverage stolen cloud credentials to target ten cloud-hosted LLMs.
One such attack involved gathering cloud credentials through a known security vulnerability but then exfiltrating these credentials and gaining access to the cloud environment, where they attempted to access local LLM models hosted by cloud providers. The attackers were able to bill the victim an astronomical amount of money for the use of the compromised LLM model.
AWS has employed automated systems that alert customers if their credentials or keys are found exposed online. However, these alerts have been criticized for doing little to stop attackers from using compromised access in this manner.
Ahl stated that it is not certain who is responsible for operating and selling these sex chat services but suspects the activity may be tied to a platform called "chub[.]ai," which offers a broad selection of pre-made AI characters with whom users can strike up conversations. The platform's website features a banner suggesting that the service is reselling access to existing cloud accounts, available for purchase starting at $5 a month.
Chub offers free registration via its website or mobile app but after a few minutes of chatting with their newfound AI friends, users are asked to pay for a subscription. Users who attempt to use the platform's services without purchasing a subscription are often met with messages implying that they should have done so.
In an interview, Ian Ahl, senior vice president of threat research at Permiso, stated that attackers are exploiting cloud credentials in order to power these illicit chatbots and evade content restrictions on AI platforms. The attack relies heavily on the attacker posing as a writer researching for a book or engaging in hypothetical scenarios to circumvent the restricted settings placed on the LLMs.
The impact of this threat cannot be overstated, as it not only poses significant security risks but also contributes to a broader uncensored AI economy that has been spurred by OpenAI and accelerated by Meta's release of its open-source Llama tool. This trend is marked by sites enabling similar AI-powered child pornographic role-play, with Chub AI offering more than 500 scenarios for these illicit activities.
Fortune profiled Chub AI in a January 2024 story describing the service as a virtual brothel advertised by illustrated girls in spaghetti strap dresses who promise a chat-based "world without feminism," where "girls offer sexual services." The founder of Chub, identified only by their handle "Lore," stated that they launched the service to help others evade content restrictions on AI platforms.
The platform has generated over $1 million in annualized revenue and is believed to be run by someone with a handle similar to Lore. While AWS initially seemed to downplay the seriousness of Permiso's research, the company has since taken steps to limit the abuse that can be committed using compromised cloud credentials.
Ahl stated that Permiso did receive multiple alerts from AWS about their exposed key, including one warning that their account may have been used by an unauthorized party. However, these restrictions placed on the exposed key did nothing to stop the attackers from using it to abuse Bedrock services.
In light of this emerging threat, organizations are being advised to ensure they are taking all necessary precautions to protect their cloud credentials and adhere to ethical guidelines when utilizing large language models on these platforms.
Related Information:
https://krebsonsecurity.com/2024/10/a-single-cloud-compromise-can-feed-an-army-of-ai-sex-bots/
Published: Thu Oct 3 08:54:54 2024 by llama3.2 3B Q4_K_M
