Hugging Face, a popular online repository for generative AI, has recently come under scrutiny from security researchers who discovered thousands of files containing hidden code that can compromise data security and steal important information, including tokens used for payment to AI and cloud operators. Security startups ProtectAI, Hiddenlayer, and Wiz have been warning about the presence of “malicious models” on the platform, with Protect AI CEO Ian Swanson stating that they found over 3,000 malicious files during a recent scan of Hugging Face.
According to Swanson, some hackers are creating fake profiles on Hugging Face to impersonate well-known technology companies like Meta, Facebook, Visa, SpaceX, and Ericsson in order to deceive users into downloading their malicious models. One such fake model, pretending to be from genomics testing startup 23AndMe, had been downloaded thousands of times before it was identified. This model contained hidden code designed to search for AWS passwords, potentially allowing the hacker to access cloud computing resources. Hugging Face promptly removed the fake 23AndMe model once the risk was identified.
In response to the security concerns, Hugging Face has integrated ProtectAI’s scanning tool into its platform to detect and warn users about any malicious code present in the models they are downloading. The company has also started verifying the profiles of major companies like OpenAI and Nvidia since 2022 and began scanning files for unsafe code in November 2021. CTO Julien Chaumond hopes that these measures, in collaboration with Protect AI and other partners, will improve trust in machine learning artifacts and make sharing and adoption easier for users.
The potential risks posed by malicious models on Hugging Face have become significant enough to prompt a joint warning from cybersecurity agencies in the United States, Canada, and Britain in April. The agencies advise businesses to thoroughly scan pre-trained models for dangerous code and run them away from critical systems to prevent any potential security breaches. Hackers targeting Hugging Face typically insert rogue instructions into the code developers download from the platform, enabling them to exploit the model when it runs without the target’s knowledge.
Hugging Face, founded by Clément Delangue, Julien Chaumond, and Thomas Wolf in 2015, has seen significant growth and success in recent years, with a valuation of $4.5 billion as of its last funding round in August 2023. Originally a teenage-focused chatbot app, the startup pivoted to become a leading platform for machine learning, earning it the nickname “Github for AI researchers.” However, as Hugging Face’s popularity continues to rise, the company is facing increased security challenges as more bad actors target the AI community. Protect AI has clarified that the number of malicious models found on Hugging Face was in the thousands, not tens of thousands.
