Multiple Security Flaws Uncovered in Open-Source AI and ML Models

October 29, 2024

Researchers have discovered more than thirty-six security vulnerabilities in several open-source artificial intelligence (AI) and machine learning (ML) models. These flaws, which could potentially enable remote code execution and data theft, were found in tools such as ChuanhuChatGPT, Lunary, and LocalAI. These vulnerabilities were reported through Protect AI's Huntr bug bounty platform.

Among the most critical vulnerabilities are two found in Lunary, a production toolkit for large language models (LLMs). Moreover, another IDOR vulnerability (CVE-2024-7473, CVSS score: 7.5) was discovered in Lunary that allows a malicious actor to modify other users' prompts by manipulating a user-controlled parameter. As Protect AI detailed in an advisory, "An attacker logs in as User A and intercepts the request to update a prompt. By modifying the 'id' parameter in the request to the 'id' of a prompt belonging to User B, the attacker can update User B's prompt without authorization."

Another critical vulnerability involves a path traversal flaw in ChuanhuChatGPT's user upload feature (CVE-2024-5982, CVSS score: 9.1) that could lead to arbitrary code execution, directory creation, and exposure of sensitive data. Two additional security flaws were identified in LocalAI, an open-source project that allows users to run self-hosted LLMs. These flaws could potentially enable malicious actors to execute arbitrary code by uploading a malicious configuration file (CVE-2024-6983, CVSS score: 8.8) and deduce valid API keys by analyzing the server's response time (CVE-2024-7010, CVSS score: 7.5). As Protect AI explained, "The vulnerability allows an attacker to perform a timing attack, which is a type of side-channel attack. By measuring the time taken to process requests with different API keys, the attacker can infer the correct API key one character at a time."

The list of vulnerabilities also includes a remote code execution flaw affecting the Deep Java Library (DJL) that stems from an arbitrary file overwrite bug in the package's untar function (CVE-2024-8396, CVSS score: 7.8). In response to these vulnerabilities, NVIDIA has released patches to fix a path traversal flaw in its NeMo generative AI framework (CVE-2024-0129, CVSS score: 6.3) that could potentially lead to code execution and data tampering. Users are urged to update their installations to the latest versions to safeguard their AI/ML supply chain and defend against potential attacks.

Protect AI has also released Vulnhuntr, an open-source Python static code analyzer that uses LLMs to detect zero-day vulnerabilities in Python codebases. This tool works by breaking the code into smaller chunks to avoid overwhelming the LLM's context window and flags potential security issues. As Dan McInerney and Marcello Salvati explained, "It automatically searches the project files for files that are likely to be the first to handle user input. Then it ingests that entire file and responds with all the potential vulnerabilities. Using this list of potential vulnerabilities, it moves on to complete the entire function call chain from user input to server output for each potential vulnerability all throughout the project one function/class at a time until it's satisfied it has the entire call chain for final analysis."

In addition to these AI framework vulnerabilities, a new jailbreak technique published by Mozilla's 0Day Investigative Network (0Din) has discovered that malicious prompts encoded in hexadecimal format and emojis could be used to bypass OpenAI ChatGPT's safeguards and create exploits for known security flaws. Security researcher Marco Figueroa explained, "The jailbreak tactic exploits a linguistic loophole by instructing the model to process a seemingly benign task: hex conversion. Since the model is optimized to follow instructions in natural language, including performing encoding or decoding tasks, it does not inherently recognize that converting hex values might produce harmful outputs. This weakness arises because the language model is designed to follow instructions step-by-step, but lacks deep context awareness to evaluate the safety of each individual step in the broader context of its ultimate goal."

Latest News

• ChatGPT Vulnerable to Hex Code Manipulation: Mozilla Report
• Fog and Akira Ransomware Operations Exploit SonicWall VPNs for Network Infiltration
• Cisco Adds Security Features to Thwart VPN Brute-Force Attacks
• Fortinet FortiManager Flaw 'FortiJump' Exploited in Zero-Day Attacks
• 'Prometei' Botnet Continues its Global Cryptojacking Campaign