Hugging Face Vulnerability Could Lead to AI Model Supply Chain Attacks

February 27, 2024

A recent report by cybersecurity researchers at HiddenLayer has highlighted a potential vulnerability in the Hugging Face Safetensors conversion service. This vulnerability could allow a threat actor to compromise the service, hijack user-submitted models, and initiate supply chain attacks.

HiddenLayer's report mentions, "It's possible to send malicious pull requests with attacker-controlled data from the Hugging Face service to any repository on the platform, as well as hijack any models that are submitted through the conversion service." This could be achieved by using a compromised model intended for conversion by the service, thereby enabling malicious actors to request changes to any repository on the platform by posing as the conversion bot.

Hugging Face is a widely-used platform that assists users in hosting pre-trained machine learning models and datasets. It also provides tools for building, deploying, and training these models. Safetensors is a format created by Hugging Face to securely store tensors, in contrast to pickles, which have been used by threat actors to execute arbitrary code and deploy Cobalt Strike, Mythic, and Metasploit stagers.

The Safetensors conversion service allows users to convert any PyTorch model (i.e., pickle) to its Safetensor equivalent through a pull request. HiddenLayer's analysis found that an attacker could theoretically compromise the hosted conversion service using a malicious PyTorch binary, thereby compromising the system hosting it. Furthermore, the token associated with SFConvertbot – an official bot designed to generate the pull request – could be extracted to send a malicious pull request to any repository on the site. This could lead to a situation where a threat actor tampers with the model and implants neural backdoors.

Researchers Eoin Wickens and Kasimir Schulz noted, "An attacker could run any arbitrary code any time someone attempted to convert their model. Without any indication to the user themselves, their models could be hijacked upon conversion." If a user attempts to convert their own private repository, the attack could lead to the theft of their Hugging Face token, access to internal models and datasets, and even the potential to poison them.

The researchers warned that an attacker could exploit the fact that any user can submit a conversion request for a public repository to hijack or alter a widely used model, creating a significant supply chain risk. They concluded, "Despite the best intentions to secure machine learning models in the Hugging Face ecosystem, the conversion service has proven to be vulnerable and has had the potential to cause a widespread supply chain attack via the Hugging Face official service."

This discovery follows the recent disclosure of LeftoverLocals (CVE-2023-4969, CVSS score: 6.5), a vulnerability that allows recovery of data from Apple, Qualcomm, AMD, and Imagination general-purpose graphics processing units (GPGPUs). This memory leak flaw, which results from a failure to adequately isolate process memory, allows a local attacker to read memory from other processes, including another user's interactive session with a large language model (LLM).

Latest News

• LiteSpeed Cache Plugin XSS Vulnerability Threatens Millions of WordPress Sites
• LockBit Ransomware Resurfaces Post Police Disruption; Threatens Greater Focus on Government Sector
• CISA Mandates Immediate Fix for ConnectWise ScreenConnect Vulnerability
• Apple Shortcuts Zero-Click Vulnerability Enables Covert Data Theft
• LockBit Ransomware Attacks Exploit ScreenConnect Servers Vulnerability