ShadowRay: Hackers Exploit Unpatched Ray Framework Vulnerability to Breach Servers

March 26, 2024

Hackers have found a way to exploit an unpatched vulnerability in Ray, a widely-used open-source AI framework, to gain unauthorized access to servers and hijack their resources. This hacking campaign, known as 'ShadowRay', has been active since at least September 5, 2023, impacting sectors such as education, cryptocurrency, and biopharma.

Ray, developed by Anyscale, is an open-source framework that allows AI and Python applications to scale across a cluster of machines for distributed computational tasks. It has over 30,500 stars on GitHub and is used globally by organizations like Amazon, Spotify, LinkedIn, Instacart, Netflix, Uber, and OpenAI for training ChatGPT.

In November 2023, Anyscale disclosed five vulnerabilities in Ray, four of which were fixed: CVE-2023-6019, CVE-2023-6020, CVE-2023-6021, and CVE-2023-48023. However, a critical remote code execution flaw, tracked as CVE-2023-48022, was left unpatched due to its lack of authentication being a long-standing design decision.

Anyscale's security advisory stated, 'The remaining CVE (CVE-2023-48022) - that Ray does not have authentication built in - is a long-standing design decision based on how Ray's security boundaries are drawn and consistent with Ray deployment best practices, though we intend to offer authentication in a future version as part of a defense-in-depth strategy.'

The company further clarified that this flaw is only exploitable in deployments that didn't adhere to the project's documentation recommendations to limit Ray's use in a strictly controlled network environment. They also stated that they do not consider these flaws as vulnerabilities but rather bugs, as their platform is designed to execute code as a distributed execution framework.

Despite the company's stance, the unpatched flaw's lack of authentication has been exploited by hackers in unsecured environments. Oligo's report reads, 'Because CVE-2023-48022 was disputed, many development teams (and most static scanning tools) are not aware that this vulnerability should concern them.'

Oligo found that hundreds of exposed Ray servers were compromised via CVE-2023-48022, with attackers gaining access to sensitive information such as AI models, environment variables, production database credentials, and cloud environment access tokens. In some cases, attackers used the compromised systems' powerful graphics cards for cryptocurrency mining operations. Others established persistence in the compromised environments using reverse shells and executed arbitrary code through Python pseudo-terminals.

After these findings, Oligo alerted many companies about the breach and provided remediation assistance. To secure Ray deployments, it is crucial to enforce firewall rules, add authorization to the Ray Dashboard port, and continuously monitor for anomalies. Default settings like binding to 0.0.0.0 should also be avoided, and tools that enhance cluster security should be used.

Related News

Latest News

Like what you see?

Get a digest of headlines, vulnerabilities, risk context, and more delivered to your inbox.

Subscribe Below

By submitting this form, you’re giving us permission to email you. You may unsubscribe at any time.

Accelerate Security Teams

Continuously identify and prioritize the risks that are most critical in your environment, and validate that your remediation efforts are reducing risk. An always-on single source-of-truth of your assets, services, and vulnerabilities.