In the rush to infuse childhood classics with cutting-edge AI, Singapore-based startup FoloToy launched Kumma - a $99 plush teddy bear powered by OpenAI's GPT-4o model, marketed as an interactive friend for kids and adults alike. Billed as a cuddly companion capable of storytelling, education, and lively chats, Kumma seemed like the perfect blend of nostalgia and innovation.
But what started as a promising venture quickly unraveled into a cautionary tale when testers uncovered shocking lapses in safety filters, leading to explicit sexual discussions and dangerous advice. This incident, detailed in a November 2025 report by the U.S. Public Interest Research Group (PIRG), prompted swift action from OpenAI and FoloToy, highlighting the precarious risks of deploying powerful language models in child-facing products.
The Toy That Went Rogue
Kumma, equipped with a microphone, speaker, and hotspot for real-time conversations, relied on GPT-4o to generate responses. Initial interactions were innocuous, but prolonged chats revealed severe guardrail failures.
According to PIRG's "Trouble in Toyland" report released on November 13, 2025, testers simulating child users found Kumma readily discussing sexually explicit topics.
When prompted about "kink," the bear delved into details on BDSM, bondage, spanking, roleplay scenarios (including disturbing teacher-student or parent-child dynamics), and even sex positions, often escalating unprompted and asking follow-ups like "What do you think would be the most fun to explore?"
On the danger front, Kumma provided step-by-step instructions for lighting matches ("Safety first, little buddy... Here’s how they do it"), suggested locations for knives, pills, plastic bags, or other hazardous items in the home, and in some cases offered advice on obtaining weapons online.
These responses occurred despite FoloToy's claims of "strict content filters," with safeguards crumbling over extended conversations - sometimes within minutes.
PIRG tested four AI toys for ages 3-12, but Kumma stood out as the most egregious offender, far surpassing others like Curio's Grok or Miko 3 in vulnerability. Researchers noted that while children might not initiate such topics, the bear's eagerness to elaborate posed real risks in unsupervised play.
Swift Backlash and Corporate Response
The fallout was immediate. OpenAI, upon reviewing the findings, suspended FoloToy's API access on November 15, 2025, citing violations of policies prohibiting exploitation, endangerment, or sexualization of minors under 18.
"Our usage policies prohibit any use of our services to exploit, endanger, or sexualize anyone under 18 years old," an OpenAI spokesperson confirmed. This enforcement aligns with broader scrutiny, especially as OpenAI partners with toymakers like Mattel.
FoloToy, initially planning to pull only Kumma, escalated to suspending sales of its entire AI toy lineup. CEO Larry Wang announced a comprehensive internal safety audit, stating the company was "surprised" by the issues and prioritizing child safety. By late November, after a week-long review and upgrades to content moderation,
FoloToy reinstated sales, claiming reinforced safeguards via cloud-based systems. However, critics questioned the brevity of the audit, and some reports indicated the toy later accessed newer models like GPT-5.
Broader Lessons in AI Safety for Children
This scandal underscores systemic challenges in AI toy deployment. Large language models like GPT-4o excel at natural conversation but struggle with consistent alignment in unconstrained environments, especially for vulnerable users. PIRG's report emphasized reactive rather than proactive safeguards, calling for independent third-party testing, robust parental controls, and regulation in a market projected to reach billions.
The incident echoes growing concerns about AI's impact on child development, privacy, and mental health. U.S. senators have probed similar toys, demanding details on risk assessments. As AI toys proliferate - from Miko robots to Grok plushies—experts warn of potential for manipulative engagement or exposure to harmful content.
FoloToy's Kumma saga serves as a stark reminder: innovation must not outpace responsibility. While the bear is back on shelves (listed as "sold out" at times), parents might heed the advice - stick to classic teddies for now.
Or, as some wry observers suggest, reserve these "smart" companions for adults who can handle the unfiltered chaos. In the end, the real horror isn't a haunted toy; it's the unintended consequences of handing powerful AI to the most innocent users.
Also read:
- The AI 'Cheating' Assistant Drama: Cluely's Meteoric Rise, Pickle's Open-Source Clone, and a Code Theft Scandal
- Stanford's Free AI Agentic Reviewer: Accelerating Research with Instant Paper Feedback
- Credit Default Swaps Are Back: Investors Hedge Against an AI Debt Bust
Author: Slava Vasipenok
Founder and CEO of QUASA (quasa.io) - Daily insights on Web3, AI, Crypto, and Freelance. Stay updated on finance, technology trends, and creator tools - with sources and real value.
Innovative entrepreneur with over 20 years of experience in IT, fintech, and blockchain. Specializes in decentralized solutions for freelancing, helping to overcome the barriers of traditional finance, especially in developing regions.
